id sid tid token lemma pos bracis-19024 1 1 on on ADP bracis-19024 1 2 the the DET bracis-19024 1 3 impact impact NOUN bracis-19024 1 4 of of ADP bracis-19024 1 5 mdp mdp NOUN bracis-19024 1 6 design design NOUN bracis-19024 1 7 for for ADP bracis-19024 1 8 reinforcement reinforcement NOUN bracis-19024 1 9 learning learning NOUN bracis-19024 1 10 agents agent NOUN bracis-19024 1 11 in in ADP bracis-19024 1 12 resource resource NOUN bracis-19024 1 13 management management NOUN bracis-19024 1 14 | | NOUN bracis-19024 1 15 springer springer NOUN bracis-19024 1 16 nature nature PROPN bracis-19024 1 17 link link PROPN bracis-19024 1 18 ( ( PUNCT bracis-19024 1 19 formerly formerly ADV bracis-19024 1 20 springerlink springerlink NOUN bracis-19024 1 21 ) ) PUNCT bracis-19024 1 22 skip skip VERB bracis-19024 1 23 to to ADP bracis-19024 1 24 main main ADJ bracis-19024 1 25 content content NOUN bracis-19024 1 26 advertisement advertisement NOUN bracis-19024 1 27 log log NOUN bracis-19024 1 28 in in ADP bracis-19024 1 29 menu menu NOUN bracis-19024 1 30 find find VERB bracis-19024 1 31 a a DET bracis-19024 1 32 journal journal NOUN bracis-19024 1 33 publish publish VERB bracis-19024 1 34 with with ADP bracis-19024 1 35 us we PRON bracis-19024 1 36 track track VERB bracis-19024 1 37 your your PRON bracis-19024 1 38 research research NOUN bracis-19024 1 39 search search NOUN bracis-19024 1 40 cart cart NOUN bracis-19024 1 41 home home NOUN bracis-19024 1 42 intelligent intelligent ADJ bracis-19024 1 43 systems system NOUN bracis-19024 1 44 conference conference NOUN bracis-19024 1 45 paper paper NOUN bracis-19024 1 46 on on ADP bracis-19024 1 47 the the DET bracis-19024 1 48 impact impact NOUN bracis-19024 1 49 of of ADP bracis-19024 1 50 mdp mdp NOUN bracis-19024 1 51 design design NOUN bracis-19024 1 52 for for ADP bracis-19024 1 53 reinforcement reinforcement NOUN bracis-19024 1 54 learning learning NOUN bracis-19024 1 55 agents agent NOUN bracis-19024 1 56 in in ADP bracis-19024 1 57 resource resource NOUN bracis-19024 1 58 management management NOUN bracis-19024 1 59 conference conference NOUN bracis-19024 1 60 paper paper NOUN bracis-19024 1 61 first first ADV bracis-19024 1 62 online online ADV bracis-19024 1 63 : : PUNCT bracis-19024 1 64 28 28 NUM bracis-19024 1 65 november november NOUN bracis-19024 1 66 2021 2021 NUM bracis-19024 2 1 pp pp ADP bracis-19024 2 2 79–93 79–93 NUM bracis-19024 2 3 cite cite VERB bracis-19024 2 4 this this DET bracis-19024 2 5 conference conference NOUN bracis-19024 2 6 paper paper NOUN bracis-19024 2 7 access access NOUN bracis-19024 2 8 provided provide VERB bracis-19024 2 9 by by ADP bracis-19024 2 10 university university PROPN bracis-19024 2 11 of of ADP bracis-19024 2 12 notre notre PROPN bracis-19024 2 13 dame dame PROPN bracis-19024 2 14 hesburgh hesburgh PROPN bracis-19024 2 15 library library PROPN bracis-19024 2 16 download download PROPN bracis-19024 2 17 book book NOUN bracis-19024 2 18 pdf pdf PROPN bracis-19024 2 19 download download NOUN bracis-19024 2 20 book book NOUN bracis-19024 2 21 epub epub PROPN bracis-19024 2 22 intelligent intelligent ADJ bracis-19024 2 23 systems system NOUN bracis-19024 2 24 ( ( PUNCT bracis-19024 2 25 bracis bracis NOUN bracis-19024 2 26 2021 2021 NUM bracis-19024 2 27 ) ) PUNCT bracis-19024 2 28 on on ADP bracis-19024 2 29 the the DET bracis-19024 2 30 impact impact NOUN bracis-19024 2 31 of of ADP bracis-19024 2 32 mdp mdp NOUN bracis-19024 2 33 design design NOUN bracis-19024 2 34 for for ADP bracis-19024 2 35 reinforcement reinforcement NOUN bracis-19024 2 36 learning learning NOUN bracis-19024 2 37 agents agent NOUN bracis-19024 2 38 in in ADP bracis-19024 2 39 resource resource NOUN bracis-19024 2 40 management management NOUN bracis-19024 2 41 download download NOUN bracis-19024 2 42 book book NOUN bracis-19024 2 43 pdf pdf PROPN bracis-19024 2 44 download download NOUN bracis-19024 2 45 book book PROPN bracis-19024 2 46 epub epub PROPN bracis-19024 2 47 renato renato PROPN bracis-19024 2 48 luiz luiz PROPN bracis-19024 2 49 de de PROPN bracis-19024 2 50 freitas freitas PROPN bracis-19024 2 51 cunha cunha NOUN bracis-19024 2 52     SPACE bracis-19024 2 53 orcid orcid NOUN bracis-19024 2 54 : : PUNCT bracis-19024 2 55 orcid.org/0000-0002-3196-300810 orcid.org/0000-0002-3196-300810 NOUN bracis-19024 2 56 & & CCONJ bracis-19024 2 57 luiz luiz PROPN bracis-19024 2 58 chaimowicz chaimowicz PROPN bracis-19024 2 59     SPACE bracis-19024 2 60 orcid orcid NOUN bracis-19024 2 61 : : PUNCT bracis-19024 2 62 orcid.org/0000-0001-8156-994110 orcid.org/0000-0001-8156-994110 NOUN bracis-19024 2 63     SPACE bracis-19024 2 64 part part NOUN bracis-19024 2 65 of of ADP bracis-19024 2 66 the the DET bracis-19024 2 67 book book NOUN bracis-19024 2 68 series series NOUN bracis-19024 2 69 : : PUNCT bracis-19024 2 70 lecture lecture NOUN bracis-19024 2 71 notes note NOUN bracis-19024 2 72 in in ADP bracis-19024 2 73 computer computer NOUN bracis-19024 2 74 science science NOUN bracis-19024 2 75 ( ( PUNCT bracis-19024 2 76 ( ( PUNCT bracis-19024 2 77 lnai lnai ADJ bracis-19024 2 78 , , PUNCT bracis-19024 2 79 volume volume NOUN bracis-19024 2 80 13073 13073 NUM bracis-19024 2 81 ) ) PUNCT bracis-19024 2 82 ) ) PUNCT bracis-19024 2 83 included include VERB bracis-19024 2 84 in in ADP bracis-19024 2 85 the the DET bracis-19024 2 86 following follow VERB bracis-19024 2 87 conference conference NOUN bracis-19024 2 88 series series NOUN bracis-19024 2 89 : : PUNCT bracis-19024 2 90 brazilian brazilian ADJ bracis-19024 2 91 conference conference NOUN bracis-19024 2 92 on on ADP bracis-19024 2 93 intelligent intelligent ADJ bracis-19024 2 94 systems system NOUN bracis-19024 2 95 761 761 NUM bracis-19024 2 96 accesses access NOUN bracis-19024 2 97 2 2 NUM bracis-19024 2 98 citations citation NOUN bracis-19024 2 99 abstract abstract ADP bracis-19024 2 100 the the DET bracis-19024 2 101 recent recent ADJ bracis-19024 2 102 progress progress NOUN bracis-19024 2 103 in in ADP bracis-19024 2 104 reinforcement reinforcement NOUN bracis-19024 2 105 learning learning NOUN bracis-19024 2 106 applications application NOUN bracis-19024 2 107 to to PART bracis-19024 2 108 resource resource NOUN bracis-19024 2 109 management management NOUN bracis-19024 2 110 presents present NOUN bracis-19024 2 111 markov markov NOUN bracis-19024 2 112 decision decision NOUN bracis-19024 2 113 processes process NOUN bracis-19024 2 114 ( ( PUNCT bracis-19024 2 115 mdps mdps NOUN bracis-19024 2 116 ) ) PUNCT bracis-19024 2 117 without without ADP bracis-19024 2 118 a a DET bracis-19024 2 119 deeper deep ADJ bracis-19024 2 120 analysis analysis NOUN bracis-19024 2 121 of of ADP bracis-19024 2 122 the the DET bracis-19024 2 123 impacts impact NOUN bracis-19024 2 124 of of ADP bracis-19024 2 125 design design NOUN bracis-19024 2 126 decisions decision NOUN bracis-19024 2 127 on on ADP bracis-19024 2 128 agent agent NOUN bracis-19024 2 129 performance performance NOUN bracis-19024 2 130 . . PUNCT bracis-19024 3 1 in in ADP bracis-19024 3 2 this this DET bracis-19024 3 3 paper paper NOUN bracis-19024 3 4 , , PUNCT bracis-19024 3 5 we we PRON bracis-19024 3 6 compare compare VERB bracis-19024 3 7 and and CCONJ bracis-19024 3 8 contrast contrast VERB bracis-19024 3 9 four four NUM bracis-19024 3 10 different different ADJ bracis-19024 3 11 mdp mdp NOUN bracis-19024 3 12 variations variation NOUN bracis-19024 3 13 , , PUNCT bracis-19024 3 14 discussing discuss VERB bracis-19024 3 15 their their PRON bracis-19024 3 16 computational computational ADJ bracis-19024 3 17 requirements requirement NOUN bracis-19024 3 18 and and CCONJ bracis-19024 3 19 impacts impact NOUN bracis-19024 3 20 on on ADP bracis-19024 3 21 agent agent NOUN bracis-19024 3 22 performance performance NOUN bracis-19024 3 23 by by ADP bracis-19024 3 24 means mean NOUN bracis-19024 3 25 of of ADP bracis-19024 3 26 an an DET bracis-19024 3 27 empirical empirical ADJ bracis-19024 3 28 analysis analysis NOUN bracis-19024 3 29 . . PUNCT bracis-19024 4 1 we we PRON bracis-19024 4 2 conclude conclude VERB bracis-19024 4 3 by by ADP bracis-19024 4 4 showing show VERB bracis-19024 4 5 that that SCONJ bracis-19024 4 6 , , PUNCT bracis-19024 4 7 in in ADP bracis-19024 4 8 our our PRON bracis-19024 4 9 experiments experiment NOUN bracis-19024 4 10 , , PUNCT bracis-19024 4 11 when when SCONJ bracis-19024 4 12 using use VERB bracis-19024 4 13 multi multi ADJ bracis-19024 4 14 - - ADJ bracis-19024 4 15 layer layer ADJ bracis-19024 4 16 perceptrons perceptron NOUN bracis-19024 4 17 as as ADP bracis-19024 4 18 approximation approximation NOUN bracis-19024 4 19 function function NOUN bracis-19024 4 20 , , PUNCT bracis-19024 4 21 a a DET bracis-19024 4 22 compact compact ADJ bracis-19024 4 23 state state NOUN bracis-19024 4 24 representation representation NOUN bracis-19024 4 25 allows allow VERB bracis-19024 4 26 transfer transfer NOUN bracis-19024 4 27 of of ADP bracis-19024 4 28 agents agent NOUN bracis-19024 4 29 between between ADP bracis-19024 4 30 environments environment NOUN bracis-19024 4 31 , , PUNCT bracis-19024 4 32 and and CCONJ bracis-19024 4 33 that that SCONJ bracis-19024 4 34 transferred transfer VERB bracis-19024 4 35 agents agent NOUN bracis-19024 4 36 have have VERB bracis-19024 4 37 good good ADJ bracis-19024 4 38 performance performance NOUN bracis-19024 4 39 and and CCONJ bracis-19024 4 40 outperform outperform ADJ bracis-19024 4 41 specialized specialized ADJ bracis-19024 4 42 agents agent NOUN bracis-19024 4 43 in in ADP bracis-19024 4 44 80 80 NUM bracis-19024 4 45 % % NOUN bracis-19024 4 46 of of ADP bracis-19024 4 47 the the DET bracis-19024 4 48 tested test VERB bracis-19024 4 49 scenarios scenario NOUN bracis-19024 4 50 , , PUNCT bracis-19024 4 51 even even ADV bracis-19024 4 52 without without ADP bracis-19024 4 53 retraining retrain VERB bracis-19024 4 54 . . PUNCT bracis-19024 5 1 access access NOUN bracis-19024 5 2 provided provide VERB bracis-19024 5 3 by by ADP bracis-19024 5 4 university university PROPN bracis-19024 5 5 of of ADP bracis-19024 5 6 notre notre PROPN bracis-19024 5 7 dame dame PROPN bracis-19024 5 8 hesburgh hesburgh PROPN bracis-19024 5 9 library library PROPN bracis-19024 5 10 . . PUNCT bracis-19024 6 1 download download PROPN bracis-19024 6 2 conference conference NOUN bracis-19024 6 3 paper paper NOUN bracis-19024 6 4 pdf pdf NOUN bracis-19024 6 5 similar similar ADJ bracis-19024 6 6 content content NOUN bracis-19024 6 7 being be AUX bracis-19024 6 8 viewed view VERB bracis-19024 6 9 by by ADP bracis-19024 6 10 others other NOUN bracis-19024 6 11 ddma ddma NOUN bracis-19024 6 12 : : PUNCT bracis-19024 6 13 discrepancy discrepancy NOUN bracis-19024 6 14 - - PUNCT bracis-19024 6 15 driven drive VERB bracis-19024 6 16 multi multi ADJ bracis-19024 6 17 - - ADJ bracis-19024 6 18 agent agent ADJ bracis-19024 6 19 reinforcement reinforcement NOUN bracis-19024 6 20 learning learning NOUN bracis-19024 6 21 chapter chapter NOUN bracis-19024 6 22 © © PROPN bracis-19024 6 23 2022 2022 NUM bracis-19024 6 24 multi multi ADJ bracis-19024 6 25 - - ADJ bracis-19024 6 26 agent agent ADJ bracis-19024 6 27 reinforcement reinforcement NOUN bracis-19024 6 28 learning learning NOUN bracis-19024 6 29 chapter chapter NOUN bracis-19024 6 30 © © PROPN bracis-19024 6 31 2020 2020 NUM bracis-19024 6 32 multi multi ADJ bracis-19024 6 33 - - ADJ bracis-19024 6 34 agent agent ADJ bracis-19024 6 35 deep deep ADJ bracis-19024 6 36 q q NOUN bracis-19024 6 37 - - PUNCT bracis-19024 6 38 learning learning NOUN bracis-19024 6 39 for for ADP bracis-19024 6 40 maintenance maintenance NOUN bracis-19024 6 41 scheduling scheduling NOUN bracis-19024 6 42 of of ADP bracis-19024 6 43 engineering engineering NOUN bracis-19024 6 44 system system NOUN bracis-19024 6 45 with with ADP bracis-19024 6 46 large large ADJ bracis-19024 6 47 - - PUNCT bracis-19024 6 48 scale scale NOUN bracis-19024 6 49 state state NOUN bracis-19024 6 50 space space NOUN bracis-19024 6 51 chapter chapter NOUN bracis-19024 6 52 © © PROPN bracis-19024 6 53 2024 2024 NUM bracis-19024 6 54 explore explore VERB bracis-19024 6 55 related relate VERB bracis-19024 6 56 subjects subject NOUN bracis-19024 6 57 discover discover VERB bracis-19024 6 58 the the DET bracis-19024 6 59 latest late ADJ bracis-19024 6 60 articles article NOUN bracis-19024 6 61 , , PUNCT bracis-19024 6 62 books book NOUN bracis-19024 6 63 and and CCONJ bracis-19024 6 64 news news NOUN bracis-19024 6 65 in in ADP bracis-19024 6 66 related related ADJ bracis-19024 6 67 subjects subject NOUN bracis-19024 6 68 , , PUNCT bracis-19024 6 69 suggested suggest VERB bracis-19024 6 70 using use VERB bracis-19024 6 71 machine machine NOUN bracis-19024 6 72 learning learning NOUN bracis-19024 6 73 . . PUNCT bracis-19024 7 1 machine machine NOUN bracis-19024 7 2 learning learn VERB bracis-19024 7 3 multiagent multiagent PROPN bracis-19024 7 4 systems system NOUN bracis-19024 7 5 performance performance NOUN bracis-19024 7 6 development development NOUN bracis-19024 7 7 stochastic stochastic NOUN bracis-19024 7 8 learning learning NOUN bracis-19024 7 9 and and CCONJ bracis-19024 7 10 adaptive adaptive ADJ bracis-19024 7 11 control control NOUN bracis-19024 7 12 artificial artificial ADJ bracis-19024 7 13 intelligence intelligence NOUN bracis-19024 7 14 agent agent NOUN bracis-19024 7 15 - - PUNCT bracis-19024 7 16 based base VERB bracis-19024 7 17 economics economic NOUN bracis-19024 7 18 1 1 NUM bracis-19024 7 19 introduction introduction NOUN bracis-19024 7 20 deep deep ADJ bracis-19024 7 21 reinforcement reinforcement NOUN bracis-19024 7 22 learning learning NOUN bracis-19024 7 23 ( ( PUNCT bracis-19024 7 24 drl drl PROPN bracis-19024 7 25 ) ) PUNCT bracis-19024 7 26 has have VERB bracis-19024 7 27 the the DET bracis-19024 7 28 potential potential NOUN bracis-19024 7 29 of of ADP bracis-19024 7 30 finding find VERB bracis-19024 7 31 novel novel ADJ bracis-19024 7 32 solutions solution NOUN bracis-19024 7 33 to to ADP bracis-19024 7 34 complex complex ADJ bracis-19024 7 35 problems problem NOUN bracis-19024 7 36 , , PUNCT bracis-19024 7 37 as as SCONJ bracis-19024 7 38 outlined outline VERB bracis-19024 7 39 by by ADP bracis-19024 7 40 recent recent ADJ bracis-19024 7 41 progress progress NOUN bracis-19024 7 42 in in ADP bracis-19024 7 43 diverse diverse ADJ bracis-19024 7 44 areas area NOUN bracis-19024 7 45 such such ADJ bracis-19024 7 46 as as ADP bracis-19024 7 47 control control NOUN bracis-19024 7 48 of of ADP bracis-19024 7 49 gene gene NOUN bracis-19024 7 50 regulatory regulatory ADJ bracis-19024 7 51 networks network NOUN bracis-19024 7 52     SPACE bracis-19024 8 1 [ [ X bracis-19024 8 2 9 9 NUM bracis-19024 8 3 ] ] PUNCT bracis-19024 8 4 , , PUNCT bracis-19024 8 5 adaptive adaptive ADJ bracis-19024 8 6 video video NOUN bracis-19024 8 7 acceleration acceleration NOUN bracis-19024 8 8     SPACE bracis-19024 9 1 [ [ X bracis-19024 9 2 11 11 NUM bracis-19024 9 3 ] ] PUNCT bracis-19024 9 4 , , PUNCT bracis-19024 9 5 and and CCONJ bracis-19024 9 6 management management NOUN bracis-19024 9 7 of of ADP bracis-19024 9 8 computational computational ADJ bracis-19024 9 9 resources resource NOUN bracis-19024 9 10     SPACE bracis-19024 10 1 [ [ X bracis-19024 10 2 7 7 NUM bracis-19024 10 3 ] ] PUNCT bracis-19024 10 4 . . PUNCT bracis-19024 11 1 resource resource NOUN bracis-19024 11 2 management management NOUN bracis-19024 11 3 , , PUNCT bracis-19024 11 4 the the DET bracis-19024 11 5 process process NOUN bracis-19024 11 6 by by ADP bracis-19024 11 7 which which PRON bracis-19024 11 8 we we PRON bracis-19024 11 9 map map VERB bracis-19024 11 10 computational computational ADJ bracis-19024 11 11 resources resource NOUN bracis-19024 11 12 to to ADP bracis-19024 11 13 the the DET bracis-19024 11 14 tasks task NOUN bracis-19024 11 15 and and CCONJ bracis-19024 11 16 jobs job NOUN bracis-19024 11 17 ( ( PUNCT bracis-19024 11 18 programs program NOUN bracis-19024 11 19 ) ) PUNCT bracis-19024 11 20 that that PRON bracis-19024 11 21 require require VERB bracis-19024 11 22 them they PRON bracis-19024 11 23 , , PUNCT bracis-19024 11 24 in in ADP bracis-19024 11 25 particular particular ADJ bracis-19024 11 26 , , PUNCT bracis-19024 11 27 is be AUX bracis-19024 11 28 an an DET bracis-19024 11 29 area area NOUN bracis-19024 11 30 in in ADP bracis-19024 11 31 which which PRON bracis-19024 11 32 recent recent ADJ bracis-19024 11 33 learning learning NOUN bracis-19024 11 34 approaches approach NOUN bracis-19024 11 35 have have AUX bracis-19024 11 36 demonstrated demonstrate VERB bracis-19024 11 37 superior superior ADJ bracis-19024 11 38 performance performance NOUN bracis-19024 11 39 over over ADP bracis-19024 11 40 classical classical ADJ bracis-19024 11 41 algorithms algorithm NOUN bracis-19024 11 42 and and CCONJ bracis-19024 11 43 optimization optimization NOUN bracis-19024 11 44 techniques technique NOUN bracis-19024 11 45 . . PUNCT bracis-19024 12 1 still still ADV bracis-19024 12 2 , , PUNCT bracis-19024 12 3 we we PRON bracis-19024 12 4 see see VERB bracis-19024 12 5 that that SCONJ bracis-19024 12 6 , , PUNCT bracis-19024 12 7 in in ADP bracis-19024 12 8 recent recent ADJ bracis-19024 12 9 work work NOUN bracis-19024 12 10 , , PUNCT bracis-19024 12 11 each each DET bracis-19024 12 12 approach approach NOUN bracis-19024 12 13 defines define VERB bracis-19024 12 14 their their PRON bracis-19024 12 15 own own ADJ bracis-19024 12 16 mdp mdp NOUN bracis-19024 12 17 formulations formulation NOUN bracis-19024 12 18 , , PUNCT bracis-19024 12 19 with with ADP bracis-19024 12 20 different different ADJ bracis-19024 12 21 design design NOUN bracis-19024 12 22 decisions decision NOUN bracis-19024 12 23 . . PUNCT bracis-19024 13 1 thus thus ADV bracis-19024 13 2 , , PUNCT bracis-19024 13 3 the the DET bracis-19024 13 4 literature literature NOUN bracis-19024 13 5 lacks lack VERB bracis-19024 13 6 an an DET bracis-19024 13 7 analysis analysis NOUN bracis-19024 13 8 of of ADP bracis-19024 13 9 the the DET bracis-19024 13 10 impact impact NOUN bracis-19024 13 11 of of ADP bracis-19024 13 12 certain certain ADJ bracis-19024 13 13 decisions decision NOUN bracis-19024 13 14 on on ADP bracis-19024 13 15 agent agent NOUN bracis-19024 13 16 performance performance NOUN bracis-19024 13 17 . . PUNCT bracis-19024 14 1 in in ADP bracis-19024 14 2 this this DET bracis-19024 14 3 paper paper NOUN bracis-19024 14 4 , , PUNCT bracis-19024 14 5 we we PRON bracis-19024 14 6 investigate investigate VERB bracis-19024 14 7 what what PRON bracis-19024 14 8 happens happen VERB bracis-19024 14 9 to to PART bracis-19024 14 10 agent agent NOUN bracis-19024 14 11 performance performance NOUN bracis-19024 14 12 as as SCONJ bracis-19024 14 13 we we PRON bracis-19024 14 14 modify modify VERB bracis-19024 14 15 an an DET bracis-19024 14 16 mdp mdp NOUN bracis-19024 14 17 , , PUNCT bracis-19024 14 18 observing observe VERB bracis-19024 14 19 the the DET bracis-19024 14 20 impacts impact NOUN bracis-19024 14 21 when when SCONJ bracis-19024 14 22 we we PRON bracis-19024 14 23 change change VERB bracis-19024 14 24 the the DET bracis-19024 14 25 state state NOUN bracis-19024 14 26 representation representation NOUN bracis-19024 14 27 , , PUNCT bracis-19024 14 28 the the DET bracis-19024 14 29 transition transition NOUN bracis-19024 14 30 function function NOUN bracis-19024 14 31 , , PUNCT bracis-19024 14 32 and and CCONJ bracis-19024 14 33 when when SCONJ bracis-19024 14 34 we we PRON bracis-19024 14 35 shape shape VERB bracis-19024 14 36 the the DET bracis-19024 14 37 reward reward NOUN bracis-19024 14 38 signal signal NOUN bracis-19024 14 39 , , PUNCT bracis-19024 14 40 performing perform VERB bracis-19024 14 41 an an DET bracis-19024 14 42 empirical empirical ADJ bracis-19024 14 43 investigation investigation NOUN bracis-19024 14 44 using use VERB bracis-19024 14 45 open open ADJ bracis-19024 14 46 - - PUNCT bracis-19024 14 47 source source NOUN bracis-19024 14 48 software software NOUN bracis-19024 14 49 from from ADP bracis-19024 14 50 the the DET bracis-19024 14 51 deep deep ADJ bracis-19024 14 52 learning learning NOUN bracis-19024 14 53 and and CCONJ bracis-19024 14 54 reinforcement reinforcement NOUN bracis-19024 14 55 learning learning NOUN bracis-19024 14 56 ( ( PUNCT bracis-19024 14 57 rl rl NOUN bracis-19024 14 58 ) ) PUNCT bracis-19024 14 59 communities community NOUN bracis-19024 14 60 . . PUNCT bracis-19024 15 1 our our PRON bracis-19024 15 2 main main ADJ bracis-19024 15 3 contribution contribution NOUN bracis-19024 15 4 is be AUX bracis-19024 15 5 the the DET bracis-19024 15 6 formulation formulation NOUN bracis-19024 15 7 and and CCONJ bracis-19024 15 8 analysis analysis NOUN bracis-19024 15 9 of of ADP bracis-19024 15 10 a a DET bracis-19024 15 11 set set NOUN bracis-19024 15 12 of of ADP bracis-19024 15 13 mdps mdps NOUN bracis-19024 15 14 derived derive VERB bracis-19024 15 15 from from ADP bracis-19024 15 16 one one NUM bracis-19024 15 17 in in ADP bracis-19024 15 18 the the DET bracis-19024 15 19 literature literature NOUN bracis-19024 15 20     SPACE bracis-19024 16 1 [ [ X bracis-19024 16 2 7 7 X bracis-19024 16 3 ] ] PUNCT bracis-19024 16 4 that that PRON bracis-19024 16 5 allows allow VERB bracis-19024 16 6 agents agent NOUN bracis-19024 16 7 to to PART bracis-19024 16 8 learn learn VERB bracis-19024 16 9 faster fast ADV bracis-19024 16 10 , , PUNCT bracis-19024 16 11 and and CCONJ bracis-19024 16 12 to to PART bracis-19024 16 13 perform perform VERB bracis-19024 16 14 transfer transfer NOUN bracis-19024 16 15 learning learning NOUN bracis-19024 16 16 with with ADP bracis-19024 16 17 various various ADJ bracis-19024 16 18 function function NOUN bracis-19024 16 19 approximation approximation NOUN bracis-19024 16 20 methods method NOUN bracis-19024 16 21 . . PUNCT bracis-19024 17 1 we we PRON bracis-19024 17 2 also also ADV bracis-19024 17 3 show show VERB bracis-19024 17 4 that that SCONJ bracis-19024 17 5 doing do VERB bracis-19024 17 6 so so ADV bracis-19024 17 7 does do AUX bracis-19024 17 8 not not PART bracis-19024 17 9 degrade degrade VERB bracis-19024 17 10 performance performance NOUN bracis-19024 17 11 in in ADP bracis-19024 17 12 the the DET bracis-19024 17 13 task task NOUN bracis-19024 17 14 . . PUNCT bracis-19024 18 1 the the DET bracis-19024 18 2 rest rest NOUN bracis-19024 18 3 of of ADP bracis-19024 18 4 this this DET bracis-19024 18 5 paper paper NOUN bracis-19024 18 6 is be AUX bracis-19024 18 7 organized organize VERB bracis-19024 18 8 as as SCONJ bracis-19024 18 9 follows follow VERB bracis-19024 18 10 : : PUNCT bracis-19024 18 11 in in ADP bracis-19024 18 12 sect sect NOUN bracis-19024 18 13 . . PUNCT bracis-19024 18 14     SPACE bracis-19024 19 1 2 2 NUM bracis-19024 19 2 , , PUNCT bracis-19024 19 3 we we PRON bracis-19024 19 4 describe describe VERB bracis-19024 19 5 the the DET bracis-19024 19 6 papers paper NOUN bracis-19024 19 7 that that PRON bracis-19024 19 8 influenced influence VERB bracis-19024 19 9 this this DET bracis-19024 19 10 one one NOUN bracis-19024 19 11 , , PUNCT bracis-19024 19 12 together together ADV bracis-19024 19 13 with with ADP bracis-19024 19 14 other other ADJ bracis-19024 19 15 deep deep ADJ bracis-19024 19 16 rl rl NOUN bracis-19024 19 17 work work NOUN bracis-19024 19 18 for for ADP bracis-19024 19 19 resource resource NOUN bracis-19024 19 20 management management NOUN bracis-19024 19 21 . . PUNCT bracis-19024 20 1 in in ADP bracis-19024 20 2 sect sect NOUN bracis-19024 20 3 . . PUNCT bracis-19024 20 4     SPACE bracis-19024 20 5 3 3 NUM bracis-19024 20 6 , , PUNCT bracis-19024 20 7 we we PRON bracis-19024 20 8 describe describe VERB bracis-19024 20 9 the the DET bracis-19024 20 10 theoretical theoretical ADJ bracis-19024 20 11 background background NOUN bracis-19024 20 12 with with ADP bracis-19024 20 13 an an DET bracis-19024 20 14 ongoing ongoing ADJ bracis-19024 20 15 example example NOUN bracis-19024 20 16 applied apply VERB bracis-19024 20 17 to to ADP bracis-19024 20 18 a a DET bracis-19024 20 19 resource resource NOUN bracis-19024 20 20 management management NOUN bracis-19024 20 21 problem problem NOUN bracis-19024 20 22 . . PUNCT bracis-19024 21 1 in in ADP bracis-19024 21 2 sect sect NOUN bracis-19024 21 3 . . PUNCT bracis-19024 21 4     SPACE bracis-19024 21 5 4 4 NUM bracis-19024 21 6 , , PUNCT bracis-19024 21 7 we we PRON bracis-19024 21 8 describe describe VERB bracis-19024 21 9 our our PRON bracis-19024 21 10 methods method NOUN bracis-19024 21 11 and and CCONJ bracis-19024 21 12 the the DET bracis-19024 21 13 proposed propose VERB bracis-19024 21 14 extensions extension NOUN bracis-19024 21 15 to to ADP bracis-19024 21 16 a a DET bracis-19024 21 17 base base ADJ bracis-19024 21 18 mdp mdp NOUN bracis-19024 21 19 . . PUNCT bracis-19024 22 1 in in ADP bracis-19024 22 2 sect sect NOUN bracis-19024 22 3 . . PUNCT bracis-19024 22 4     SPACE bracis-19024 22 5 5 5 NUM bracis-19024 22 6 , , PUNCT bracis-19024 22 7 we we PRON bracis-19024 22 8 describe describe VERB bracis-19024 22 9 our our PRON bracis-19024 22 10 experimental experimental ADJ bracis-19024 22 11 framework framework NOUN bracis-19024 22 12 , , PUNCT bracis-19024 22 13 along along ADP bracis-19024 22 14 with with ADP bracis-19024 22 15 the the DET bracis-19024 22 16 experiments experiment NOUN bracis-19024 22 17 designed design VERB bracis-19024 22 18 to to PART bracis-19024 22 19 evaluate evaluate VERB bracis-19024 22 20 our our PRON bracis-19024 22 21 mdps mdps NOUN bracis-19024 22 22 . . PUNCT bracis-19024 23 1 in in ADP bracis-19024 23 2 sect sect NOUN bracis-19024 23 3 . . PUNCT bracis-19024 23 4     SPACE bracis-19024 23 5 6 6 NUM bracis-19024 23 6 we we PRON bracis-19024 23 7 present present VERB bracis-19024 23 8 our our PRON bracis-19024 23 9 concluding concluding NOUN bracis-19024 23 10 remarks remark NOUN bracis-19024 23 11 and and CCONJ bracis-19024 23 12 a a DET bracis-19024 23 13 brief brief ADJ bracis-19024 23 14 discussion discussion NOUN bracis-19024 23 15 of of ADP bracis-19024 23 16 consequences consequence NOUN bracis-19024 23 17 of of ADP bracis-19024 23 18 the the DET bracis-19024 23 19 work work NOUN bracis-19024 23 20 described describe VERB bracis-19024 23 21 here here ADV bracis-19024 23 22 . . PUNCT bracis-19024 24 1 2 2 NUM bracis-19024 24 2 related relate VERB bracis-19024 24 3 work work NOUN bracis-19024 24 4 in in ADP bracis-19024 24 5 recent recent ADJ bracis-19024 24 6 years year NOUN bracis-19024 24 7 , , PUNCT bracis-19024 24 8 interest interest NOUN bracis-19024 24 9 in in ADP bracis-19024 24 10 deep deep ADJ bracis-19024 24 11 reinforcement reinforcement NOUN bracis-19024 24 12 learning learning NOUN bracis-19024 24 13 ( ( PUNCT bracis-19024 24 14 rl rl NOUN bracis-19024 24 15 ) ) PUNCT bracis-19024 24 16 applied apply VERB bracis-19024 24 17 to to ADP bracis-19024 24 18 executing execute VERB bracis-19024 24 19 the the DET bracis-19024 24 20 scheduling scheduling NOUN bracis-19024 24 21 of of ADP bracis-19024 24 22 computing computing NOUN bracis-19024 24 23 jobs job NOUN bracis-19024 24 24 was be AUX bracis-19024 24 25 probably probably ADV bracis-19024 24 26 inspired inspire VERB bracis-19024 24 27 by by ADP bracis-19024 24 28 the the DET bracis-19024 24 29 deep deep ADJ bracis-19024 24 30 resource resource NOUN bracis-19024 24 31 management management NOUN bracis-19024 24 32 ( ( PUNCT bracis-19024 24 33 deeprm deeprm PROPN bracis-19024 24 34 ) ) PUNCT bracis-19024 24 35     SPACE bracis-19024 25 1 [ [ X bracis-19024 25 2 7 7 X bracis-19024 25 3 ] ] ADJ bracis-19024 25 4 agent agent NOUN bracis-19024 25 5 and and CCONJ bracis-19024 25 6 environment environment NOUN bracis-19024 25 7 . . PUNCT bracis-19024 26 1 deeprm deeprm PROPN bracis-19024 26 2 presented present VERB bracis-19024 26 3 an an DET bracis-19024 26 4 approach approach NOUN bracis-19024 26 5 of of ADP bracis-19024 26 6 using use VERB bracis-19024 26 7 policy policy NOUN bracis-19024 26 8 gradients gradient NOUN bracis-19024 26 9 to to PART bracis-19024 26 10 schedule schedule VERB bracis-19024 26 11 jobs job NOUN bracis-19024 26 12 based base VERB bracis-19024 26 13 on on ADP bracis-19024 26 14 cpu cpu NOUN bracis-19024 26 15 and and CCONJ bracis-19024 26 16 memory memory NOUN bracis-19024 26 17 requirements requirement NOUN bracis-19024 26 18 . . PUNCT bracis-19024 27 1 deeprm deeprm PROPN bracis-19024 27 2 ’s ’s PART bracis-19024 27 3 approach approach NOUN bracis-19024 27 4 uses use VERB bracis-19024 27 5 images image NOUN bracis-19024 27 6 to to PART bracis-19024 27 7 represent represent VERB bracis-19024 27 8 jobs job NOUN bracis-19024 27 9 , , PUNCT bracis-19024 27 10 and and CCONJ bracis-19024 27 11 a a DET bracis-19024 27 12 window window NOUN bracis-19024 27 13 of of ADP bracis-19024 27 14 jobs job NOUN bracis-19024 27 15 from from ADP bracis-19024 27 16 which which PRON bracis-19024 27 17 it it PRON bracis-19024 27 18 can can AUX bracis-19024 27 19 choose choose VERB bracis-19024 27 20 which which DET bracis-19024 27 21 job job NOUN bracis-19024 27 22 to to PART bracis-19024 27 23 schedule schedule VERB bracis-19024 27 24 next next ADV bracis-19024 27 25 . . PUNCT bracis-19024 28 1 it it PRON bracis-19024 28 2 was be AUX bracis-19024 28 3 shown show VERB bracis-19024 28 4 that that SCONJ bracis-19024 28 5 deeprm deeprm PROPN bracis-19024 28 6 can can AUX bracis-19024 28 7 learn learn VERB bracis-19024 28 8 to to PART bracis-19024 28 9 schedule schedule VERB bracis-19024 28 10 based base VERB bracis-19024 28 11 on on ADP bracis-19024 28 12 different different ADJ bracis-19024 28 13 metrics metric NOUN bracis-19024 28 14 . . PUNCT bracis-19024 29 1 domeniconi domeniconi PROPN bracis-19024 29 2 et et PROPN bracis-19024 29 3 al al PROPN bracis-19024 29 4 . . PUNCT bracis-19024 30 1 [ [ X bracis-19024 30 2 3 3 X bracis-19024 30 3 ] ] PUNCT bracis-19024 30 4 proposed propose VERB bracis-19024 30 5 cush cush NOUN bracis-19024 30 6 , , PUNCT bracis-19024 30 7 a a DET bracis-19024 30 8 system system NOUN bracis-19024 30 9 that that PRON bracis-19024 30 10 built build VERB bracis-19024 30 11 on on ADP bracis-19024 30 12 deeprm deeprm NOUN bracis-19024 30 13 to to PART bracis-19024 30 14 schedule schedule VERB bracis-19024 30 15 for for ADP bracis-19024 30 16 cpus cpus NOUN bracis-19024 30 17 and and CCONJ bracis-19024 30 18 gpus gpu NOUN bracis-19024 30 19 , , PUNCT bracis-19024 30 20 but but CCONJ bracis-19024 30 21 proposed propose VERB bracis-19024 30 22 a a DET bracis-19024 30 23 hierarchical hierarchical ADJ bracis-19024 30 24 agent agent NOUN bracis-19024 30 25 by by ADP bracis-19024 30 26 introducing introduce VERB bracis-19024 30 27 an an DET bracis-19024 30 28 additional additional ADJ bracis-19024 30 29 convolutional convolutional ADJ bracis-19024 30 30 neural neural ADJ bracis-19024 30 31 network network NOUN bracis-19024 30 32 ( ( PUNCT bracis-19024 30 33 cnn cnn PROPN bracis-19024 30 34 ) ) PUNCT bracis-19024 30 35 that that PRON bracis-19024 30 36 chooses choose VERB bracis-19024 30 37 which which DET bracis-19024 30 38 job job NOUN bracis-19024 30 39 is be AUX bracis-19024 30 40 going go VERB bracis-19024 30 41 to to PART bracis-19024 30 42 be be AUX bracis-19024 30 43 scheduled schedule VERB bracis-19024 30 44 next next ADV bracis-19024 30 45 , , PUNCT bracis-19024 30 46 and and CCONJ bracis-19024 30 47 then then ADV bracis-19024 30 48 uses use VERB bracis-19024 30 49 a a DET bracis-19024 30 50 policy policy NOUN bracis-19024 30 51 network network NOUN bracis-19024 30 52 to to PART bracis-19024 30 53 choose choose VERB bracis-19024 30 54 the the DET bracis-19024 30 55 scheduling scheduling NOUN bracis-19024 30 56 policy policy NOUN bracis-19024 30 57 to to PART bracis-19024 30 58 use use VERB bracis-19024 30 59 with with ADP bracis-19024 30 60 the the DET bracis-19024 30 61 previously previously ADV bracis-19024 30 62 selected select VERB bracis-19024 30 63 job job NOUN bracis-19024 30 64 . . PUNCT bracis-19024 31 1 it it PRON bracis-19024 31 2 is be AUX bracis-19024 31 3 important important ADJ bracis-19024 31 4 to to PART bracis-19024 31 5 highlight highlight VERB bracis-19024 31 6 a a DET bracis-19024 31 7 major major ADJ bracis-19024 31 8 difference difference NOUN bracis-19024 31 9 between between ADP bracis-19024 31 10 deeprm deeprm NOUN bracis-19024 31 11 and and CCONJ bracis-19024 31 12 cush cush NOUN bracis-19024 31 13 : : PUNCT bracis-19024 31 14 whereas whereas SCONJ bracis-19024 31 15 deeprm deeprm NOUN bracis-19024 31 16 learns learn VERB bracis-19024 31 17 the the DET bracis-19024 31 18 scheduling scheduling NOUN bracis-19024 31 19 policy policy NOUN bracis-19024 31 20 itself itself PRON bracis-19024 31 21 , , PUNCT bracis-19024 31 22 cush cush NOUN bracis-19024 31 23 is be AUX bracis-19024 31 24 essentially essentially ADV bracis-19024 31 25 a a DET bracis-19024 31 26 classifier classifier NOUN bracis-19024 31 27 , , PUNCT bracis-19024 31 28 which which PRON bracis-19024 31 29 chooses choose VERB bracis-19024 31 30 between between ADP bracis-19024 31 31 two two NUM bracis-19024 31 32 existing exist VERB bracis-19024 31 33 policies policy NOUN bracis-19024 31 34 . . PUNCT bracis-19024 32 1 this this PRON bracis-19024 32 2 means mean VERB bracis-19024 32 3 that that SCONJ bracis-19024 32 4 , , PUNCT bracis-19024 32 5 even even ADV bracis-19024 32 6 without without ADP bracis-19024 32 7 training training NOUN bracis-19024 32 8 , , PUNCT bracis-19024 32 9 cush cush NOUN bracis-19024 32 10 ’s ’s PART bracis-19024 32 11 behavior behavior NOUN bracis-19024 32 12 is be AUX bracis-19024 32 13 more more ADV bracis-19024 32 14 stable stable ADJ bracis-19024 32 15 than than ADP bracis-19024 32 16 that that PRON bracis-19024 32 17 of of ADP bracis-19024 32 18 deeprm deeprm NOUN bracis-19024 32 19 , , PUNCT bracis-19024 32 20 since since SCONJ bracis-19024 32 21 deeprm deeprm ADJ bracis-19024 32 22 - - PUNCT bracis-19024 32 23 style style NOUN bracis-19024 32 24 schedulers scheduler NOUN bracis-19024 32 25 might might AUX bracis-19024 32 26 get get AUX bracis-19024 32 27 stuck stick VERB bracis-19024 32 28 in in ADP bracis-19024 32 29 local local ADJ bracis-19024 32 30 minima minima NOUN bracis-19024 32 31 , , PUNCT bracis-19024 32 32 as as SCONJ bracis-19024 32 33 reported report VERB bracis-19024 32 34 by by ADP bracis-19024 32 35     SPACE bracis-19024 32 36 de de PROPN bracis-19024 32 37 freitas freitas PROPN bracis-19024 32 38 cunha cunha NOUN bracis-19024 32 39 and and CCONJ bracis-19024 32 40 chaimowicz chaimowicz NOUN bracis-19024 33 1 [ [ X bracis-19024 33 2 2 2 NUM bracis-19024 33 3 ] ] PUNCT bracis-19024 33 4 , , PUNCT bracis-19024 34 1 who who PRON bracis-19024 34 2 investigated investigate VERB bracis-19024 34 3 the the DET bracis-19024 34 4 behavior behavior NOUN bracis-19024 34 5 of of ADP bracis-19024 34 6 deeprm deeprm ADJ bracis-19024 34 7 - - PUNCT bracis-19024 34 8 style style NOUN bracis-19024 34 9 agents agent NOUN bracis-19024 34 10 when when SCONJ bracis-19024 34 11 trained train VERB bracis-19024 34 12 with with ADP bracis-19024 34 13 state state NOUN bracis-19024 34 14 - - PUNCT bracis-19024 34 15 of of ADP bracis-19024 34 16 - - PUNCT bracis-19024 34 17 the the DET bracis-19024 34 18 - - PUNCT bracis-19024 34 19 art art NOUN bracis-19024 34 20 rl rl NOUN bracis-19024 34 21 algorithms algorithm NOUN bracis-19024 34 22 such such ADJ bracis-19024 34 23 as as ADP bracis-19024 34 24 advantage advantage NOUN bracis-19024 34 25 actor actor NOUN bracis-19024 34 26 - - PUNCT bracis-19024 34 27 critic critic NOUN bracis-19024 34 28 ( ( PUNCT bracis-19024 34 29 a2c a2c PROPN bracis-19024 34 30 ) ) PUNCT bracis-19024 34 31 and and CCONJ bracis-19024 34 32 proximal proximal ADJ bracis-19024 34 33 policy policy NOUN bracis-19024 34 34 optimization optimization NOUN bracis-19024 34 35 ( ( PUNCT bracis-19024 34 36 ppo ppo PROPN bracis-19024 34 37 ) ) PUNCT bracis-19024 34 38 , , PUNCT bracis-19024 34 39 and and CCONJ bracis-19024 34 40 proposed propose VERB bracis-19024 34 41 an an DET bracis-19024 34 42 openai openai ADJ bracis-19024 34 43 gym gym NOUN bracis-19024 34 44 environment environment NOUN bracis-19024 34 45 for for ADP bracis-19024 34 46 easier easy ADJ bracis-19024 34 47 evaluation evaluation NOUN bracis-19024 34 48 of of ADP bracis-19024 34 49 rl rl X bracis-19024 34 50 agents agent NOUN bracis-19024 34 51 for for ADP bracis-19024 34 52 job job NOUN bracis-19024 34 53 scheduling scheduling NOUN bracis-19024 34 54 . . PUNCT bracis-19024 35 1 another another DET bracis-19024 35 2 agent agent NOUN bracis-19024 35 3 that that PRON bracis-19024 35 4 has have AUX bracis-19024 35 5 been be AUX bracis-19024 35 6 proposed propose VERB bracis-19024 35 7 recently recently ADV bracis-19024 35 8 , , PUNCT bracis-19024 35 9 and and CCONJ bracis-19024 35 10 that that PRON bracis-19024 35 11 learns learn VERB bracis-19024 35 12 the the DET bracis-19024 35 13 scheduling scheduling NOUN bracis-19024 35 14 policy policy NOUN bracis-19024 35 15 itself itself PRON bracis-19024 35 16 , , PUNCT bracis-19024 35 17 is be AUX bracis-19024 35 18 rlscheduler rlscheduler NOUN bracis-19024 35 19     SPACE bracis-19024 36 1 [ [ X bracis-19024 36 2 16 16 NUM bracis-19024 36 3 ] ] PUNCT bracis-19024 36 4 . . PUNCT bracis-19024 37 1 rlscheduler rlscheduler NOUN bracis-19024 37 2 is be AUX bracis-19024 37 3 a a DET bracis-19024 37 4 ppo ppo PROPN bracis-19024 37 5 - - PUNCT bracis-19024 37 6 based base VERB bracis-19024 37 7 agent agent NOUN bracis-19024 37 8 with with ADP bracis-19024 37 9 a a DET bracis-19024 37 10 fully fully ADV bracis-19024 37 11 convolutional convolutional ADJ bracis-19024 37 12 neural neural ADJ bracis-19024 37 13 network network NOUN bracis-19024 37 14 for for ADP bracis-19024 37 15 scoring score VERB bracis-19024 37 16 jobs job NOUN bracis-19024 37 17 in in ADP bracis-19024 37 18 a a DET bracis-19024 37 19 fixed fix VERB bracis-19024 37 20 window window NOUN bracis-19024 37 21 of of ADP bracis-19024 37 22 size size NOUN bracis-19024 37 23 128 128 NUM bracis-19024 37 24 . . PUNCT bracis-19024 38 1 the the DET bracis-19024 38 2 major major ADJ bracis-19024 38 3 innovation innovation NOUN bracis-19024 38 4 in in ADP bracis-19024 38 5 rlscheduler rlscheduler NOUN bracis-19024 38 6 is be AUX bracis-19024 38 7 in in ADP bracis-19024 38 8 the the DET bracis-19024 38 9 training training NOUN bracis-19024 38 10 setting setting NOUN bracis-19024 38 11 , , PUNCT bracis-19024 38 12 in in ADP bracis-19024 38 13 which which PRON bracis-19024 38 14 the the DET bracis-19024 38 15 authors author NOUN bracis-19024 38 16 combine combine VERB bracis-19024 38 17 synthetic synthetic ADJ bracis-19024 38 18 workload workload NOUN bracis-19024 38 19 traces trace NOUN bracis-19024 38 20 with with ADP bracis-19024 38 21 real real ADJ bracis-19024 38 22 workload workload NOUN bracis-19024 38 23 traces trace NOUN bracis-19024 38 24 to to PART bracis-19024 38 25 present present VERB bracis-19024 38 26 the the DET bracis-19024 38 27 agent agent NOUN bracis-19024 38 28 with with ADP bracis-19024 38 29 ever ever ADV bracis-19024 38 30 more more ADV bracis-19024 38 31 difficult difficult ADJ bracis-19024 38 32 settings setting NOUN bracis-19024 38 33 , , PUNCT bracis-19024 38 34 similar similar ADJ bracis-19024 38 35 to to ADP bracis-19024 38 36 learning learn VERB bracis-19024 38 37 a a DET bracis-19024 38 38 curriculum curriculum NOUN bracis-19024 38 39 of of ADP bracis-19024 38 40 tasks task NOUN bracis-19024 38 41 . . PUNCT bracis-19024 39 1 similarly similarly ADV bracis-19024 39 2 to to PART bracis-19024 39 3 cush cush VERB bracis-19024 39 4 , , PUNCT bracis-19024 39 5 other other ADJ bracis-19024 39 6 agents agent NOUN bracis-19024 39 7 that that PRON bracis-19024 39 8 use use VERB bracis-19024 39 9 a a DET bracis-19024 39 10 classification classification NOUN bracis-19024 39 11 approach approach NOUN bracis-19024 39 12 have have AUX bracis-19024 39 13 been be AUX bracis-19024 39 14 proposed propose VERB bracis-19024 39 15 . . PUNCT bracis-19024 40 1 a a DET bracis-19024 40 2 recent recent ADJ bracis-19024 40 3 one one NOUN bracis-19024 40 4 is be AUX bracis-19024 40 5 the the DET bracis-19024 40 6 deep deep ADJ bracis-19024 40 7 reinforcement reinforcement NOUN bracis-19024 40 8 agent agent NOUN bracis-19024 40 9 for for ADP bracis-19024 40 10 scheduling scheduling NOUN bracis-19024 40 11 in in ADP bracis-19024 40 12 hpc hpc PROPN bracis-19024 40 13 ( ( PUNCT bracis-19024 40 14 dras dra NOUN bracis-19024 40 15 ) ) PUNCT bracis-19024 40 16     SPACE bracis-19024 41 1 [ [ X bracis-19024 41 2 4 4 NUM bracis-19024 41 3 ] ] PUNCT bracis-19024 41 4 , , PUNCT bracis-19024 41 5 which which PRON bracis-19024 41 6 classifies classify VERB bracis-19024 41 7 jobs job NOUN bracis-19024 41 8 in in ADP bracis-19024 41 9 three three NUM bracis-19024 41 10 categories category NOUN bracis-19024 41 11 : : PUNCT bracis-19024 41 12 ready ready ADJ bracis-19024 41 13 , , PUNCT bracis-19024 41 14 reserved reserved ADJ bracis-19024 41 15 , , PUNCT bracis-19024 41 16 and and CCONJ bracis-19024 41 17 backfilled backfille VERB bracis-19024 41 18 . . PUNCT bracis-19024 42 1 after after ADP bracis-19024 42 2 this this DET bracis-19024 42 3 classification classification NOUN bracis-19024 42 4 step step NOUN bracis-19024 42 5 , , PUNCT bracis-19024 42 6 the the DET bracis-19024 42 7 cluster cluster NOUN bracis-19024 42 8 scheduler scheduler NOUN bracis-19024 42 9 takes take VERB bracis-19024 42 10 the the DET bracis-19024 42 11 output output NOUN bracis-19024 42 12 of of ADP bracis-19024 42 13 this this DET bracis-19024 42 14 classification classification NOUN bracis-19024 42 15 and and CCONJ bracis-19024 42 16 allocates allocate VERB bracis-19024 42 17 jobs job NOUN bracis-19024 42 18 accordingly accordingly ADV bracis-19024 42 19 ( ( PUNCT bracis-19024 42 20 for for ADP bracis-19024 42 21 example example NOUN bracis-19024 42 22 , , PUNCT bracis-19024 42 23 by by ADP bracis-19024 42 24 reserving reserve VERB bracis-19024 42 25 slots slot NOUN bracis-19024 42 26 in in ADP bracis-19024 42 27 the the DET bracis-19024 42 28 future future NOUN bracis-19024 42 29 for for ADP bracis-19024 42 30 reserved reserved ADJ bracis-19024 42 31 jobs job NOUN bracis-19024 42 32 , , PUNCT bracis-19024 42 33 scheduling schedule VERB bracis-19024 42 34 immediately immediately ADV bracis-19024 42 35 ready ready ADJ bracis-19024 42 36 jobs job NOUN bracis-19024 42 37 , , PUNCT bracis-19024 42 38 and and CCONJ bracis-19024 42 39 finding find VERB bracis-19024 42 40 “ " PUNCT bracis-19024 42 41 holes hole NOUN bracis-19024 42 42 ” " PUNCT bracis-19024 42 43 in in ADP bracis-19024 42 44 the the DET bracis-19024 42 45 schedule schedule NOUN bracis-19024 42 46 for for ADP bracis-19024 42 47 backfilled backfille VERB bracis-19024 42 48 jobs job NOUN bracis-19024 42 49 ) ) PUNCT bracis-19024 42 50 . . PUNCT bracis-19024 43 1 dras dra NOUN bracis-19024 43 2 uses use VERB bracis-19024 43 3 a a DET bracis-19024 43 4 five five NUM bracis-19024 43 5 - - PUNCT bracis-19024 43 6 layer layer NOUN bracis-19024 43 7 cnn cnn NOUN bracis-19024 43 8 that that PRON bracis-19024 43 9 works work VERB bracis-19024 43 10 in in ADP bracis-19024 43 11 two two NUM bracis-19024 43 12 levels level NOUN bracis-19024 43 13 , , PUNCT bracis-19024 43 14 with with ADP bracis-19024 43 15 the the DET bracis-19024 43 16 first first ADJ bracis-19024 43 17 level level NOUN bracis-19024 43 18 selecting select VERB bracis-19024 43 19 jobs job NOUN bracis-19024 43 20 for for ADP bracis-19024 43 21 immediate immediate ADJ bracis-19024 43 22 and and CCONJ bracis-19024 43 23 reserved reserved ADJ bracis-19024 43 24 execution execution NOUN bracis-19024 43 25 , , PUNCT bracis-19024 43 26 and and CCONJ bracis-19024 43 27 the the DET bracis-19024 43 28 second second ADJ bracis-19024 43 29 layer layer NOUN bracis-19024 43 30 for for ADP bracis-19024 43 31 backfilled backfille VERB bracis-19024 43 32 execution execution NOUN bracis-19024 43 33 . . PUNCT bracis-19024 44 1 none none NOUN bracis-19024 44 2 of of ADP bracis-19024 44 3 the the DET bracis-19024 44 4 papers paper NOUN bracis-19024 44 5 mentioned mention VERB bracis-19024 44 6 above above ADP bracis-19024 44 7 discuss discuss VERB bracis-19024 44 8 the the DET bracis-19024 44 9 impact impact NOUN bracis-19024 44 10 of of ADP bracis-19024 44 11 their their PRON bracis-19024 44 12 design design NOUN bracis-19024 44 13 decisions decision NOUN bracis-19024 44 14 , , PUNCT bracis-19024 44 15 resorting resort VERB bracis-19024 44 16 to to ADP bracis-19024 44 17 only only ADV bracis-19024 44 18 comparing compare VERB bracis-19024 44 19 their their PRON bracis-19024 44 20 results result NOUN bracis-19024 44 21 with with ADP bracis-19024 44 22 existing exist VERB bracis-19024 44 23 algorithms algorithm NOUN bracis-19024 44 24 . . PUNCT bracis-19024 45 1 in in ADP bracis-19024 45 2 this this DET bracis-19024 45 3 paper paper NOUN bracis-19024 45 4 , , PUNCT bracis-19024 45 5 we we PRON bracis-19024 45 6 aim aim VERB bracis-19024 45 7 to to PART bracis-19024 45 8 analyze analyze VERB bracis-19024 45 9 how how SCONJ bracis-19024 45 10 decisions decision NOUN bracis-19024 45 11 in in ADP bracis-19024 45 12 mdp mdp NOUN bracis-19024 45 13 design design NOUN bracis-19024 45 14 impact impact NOUN bracis-19024 45 15 deeprm deeprm NOUN bracis-19024 45 16 - - PUNCT bracis-19024 45 17 style style NOUN bracis-19024 45 18 algorithms algorithm NOUN bracis-19024 45 19 , , PUNCT bracis-19024 45 20 and and CCONJ bracis-19024 45 21 how how SCONJ bracis-19024 45 22 they they PRON bracis-19024 45 23 impact impact VERB bracis-19024 45 24 both both DET bracis-19024 45 25 computational computational ADJ bracis-19024 45 26 performance performance NOUN bracis-19024 45 27 and and CCONJ bracis-19024 45 28 scheduling scheduling NOUN bracis-19024 45 29 performance performance NOUN bracis-19024 45 30 . . PUNCT bracis-19024 46 1 3 3 NUM bracis-19024 46 2 background background NOUN bracis-19024 46 3 in in ADP bracis-19024 46 4 this this DET bracis-19024 46 5 section section NOUN bracis-19024 46 6 , , PUNCT bracis-19024 46 7 we we PRON bracis-19024 46 8 describe describe VERB bracis-19024 46 9 the the DET bracis-19024 46 10 background background NOUN bracis-19024 46 11 needed need VERB bracis-19024 46 12 to to PART bracis-19024 46 13 understand understand VERB bracis-19024 46 14 the the DET bracis-19024 46 15 techniques technique NOUN bracis-19024 46 16 and and CCONJ bracis-19024 46 17 methodology methodology NOUN bracis-19024 46 18 presented present VERB bracis-19024 46 19 in in ADP bracis-19024 46 20 this this DET bracis-19024 46 21 paper paper NOUN bracis-19024 46 22 . . PUNCT bracis-19024 47 1 to to PART bracis-19024 47 2 help help VERB bracis-19024 47 3 in in ADP bracis-19024 47 4 understanding understanding NOUN bracis-19024 47 5 , , PUNCT bracis-19024 47 6 we we PRON bracis-19024 47 7 will will AUX bracis-19024 47 8 use use VERB bracis-19024 47 9 a a DET bracis-19024 47 10 resource resource NOUN bracis-19024 47 11 management management NOUN bracis-19024 47 12 problem problem NOUN bracis-19024 47 13 as as ADP bracis-19024 47 14 running run VERB bracis-19024 47 15 example example NOUN bracis-19024 47 16 throughout throughout ADP bracis-19024 47 17 this this DET bracis-19024 47 18 section section NOUN bracis-19024 47 19 . . PUNCT bracis-19024 48 1 3.1 3.1 NUM bracis-19024 48 2 batch batch NOUN bracis-19024 48 3 job job NOUN bracis-19024 48 4 scheduling scheduling NOUN bracis-19024 48 5 the the DET bracis-19024 48 6 primary primary ADJ bracis-19024 48 7 goal goal NOUN bracis-19024 48 8 of of ADP bracis-19024 48 9 a a DET bracis-19024 48 10 job job NOUN bracis-19024 48 11 scheduler scheduler NOUN bracis-19024 48 12 is be AUX bracis-19024 48 13 to to PART bracis-19024 48 14 manage manage VERB bracis-19024 48 15 the the DET bracis-19024 48 16 job job NOUN bracis-19024 48 17 queue queue NOUN bracis-19024 48 18 and and CCONJ bracis-19024 48 19 coordinate coordinate VERB bracis-19024 48 20 execution execution NOUN bracis-19024 48 21 of of ADP bracis-19024 48 22 jobs job NOUN bracis-19024 48 23 in in ADP bracis-19024 48 24 high high ADJ bracis-19024 48 25 performance performance NOUN bracis-19024 48 26 computing computing NOUN bracis-19024 48 27 ( ( PUNCT bracis-19024 48 28 hpc hpc NOUN bracis-19024 48 29 ) ) PUNCT bracis-19024 48 30 clusters cluster NOUN bracis-19024 48 31 , , PUNCT bracis-19024 48 32 while while SCONJ bracis-19024 48 33 matching match VERB bracis-19024 48 34 jobs job NOUN bracis-19024 48 35 to to ADP bracis-19024 48 36 resources resource NOUN bracis-19024 48 37 in in ADP bracis-19024 48 38 an an DET bracis-19024 48 39 efficient efficient ADJ bracis-19024 48 40 way way NOUN bracis-19024 48 41 . . PUNCT bracis-19024 49 1 in in ADP bracis-19024 49 2 a a DET bracis-19024 49 3 discrete discrete ADJ bracis-19024 49 4 time time NOUN bracis-19024 49 5 setting setting NOUN bracis-19024 49 6 , , PUNCT bracis-19024 49 7 at at ADP bracis-19024 49 8 each each DET bracis-19024 49 9 time time NOUN bracis-19024 49 10 step step NOUN bracis-19024 49 11 , , PUNCT bracis-19024 49 12 zero zero NUM bracis-19024 49 13 or or CCONJ bracis-19024 49 14 more more ADJ bracis-19024 49 15 jobs job NOUN bracis-19024 49 16 may may AUX bracis-19024 49 17 arrive arrive VERB bracis-19024 49 18 in in ADP bracis-19024 49 19 the the DET bracis-19024 49 20 queue queue NOUN bracis-19024 49 21 for for ADP bracis-19024 49 22 processing processing NOUN bracis-19024 49 23 , , PUNCT bracis-19024 49 24 and and CCONJ bracis-19024 49 25 the the DET bracis-19024 49 26 scheduler scheduler NOUN bracis-19024 49 27 ’s ’s PART bracis-19024 49 28 job job NOUN bracis-19024 49 29 is be AUX bracis-19024 49 30 to to PART bracis-19024 49 31 allocate allocate VERB bracis-19024 49 32 jobs job NOUN bracis-19024 49 33 to to ADP bracis-19024 49 34 resources resource NOUN bracis-19024 49 35 while while SCONJ bracis-19024 49 36 satisfying satisfy VERB bracis-19024 49 37 their their PRON bracis-19024 49 38 resource resource NOUN bracis-19024 49 39 requirements requirement NOUN bracis-19024 49 40 . . PUNCT bracis-19024 50 1 the the DET bracis-19024 50 2 job job NOUN bracis-19024 50 3 scheduler scheduler NOUN bracis-19024 50 4 guarantees guarantee VERB bracis-19024 50 5 jobs job NOUN bracis-19024 50 6 execute execute VERB bracis-19024 50 7 when when SCONJ bracis-19024 50 8 requested request VERB bracis-19024 50 9 resources resource NOUN bracis-19024 50 10 are be AUX bracis-19024 50 11 available available ADJ bracis-19024 50 12 , , PUNCT bracis-19024 50 13 and and CCONJ bracis-19024 50 14 usually usually ADV bracis-19024 50 15 guarantee guarantee VERB bracis-19024 50 16 there there PRON bracis-19024 50 17 wo will AUX bracis-19024 50 18 n’t not PART bracis-19024 50 19 be be AUX bracis-19024 50 20 oversubscription oversubscription NOUN bracis-19024 50 21 of of ADP bracis-19024 50 22 resourcesfootnote resourcesfootnote PROPN bracis-19024 50 23 1 1 NUM bracis-19024 50 24 . . PUNCT bracis-19024 51 1 given give VERB bracis-19024 51 2 this this DET bracis-19024 51 3 primary primary ADJ bracis-19024 51 4 goal goal NOUN bracis-19024 51 5 , , PUNCT bracis-19024 51 6 secondary secondary ADJ bracis-19024 51 7 goals goal NOUN bracis-19024 51 8 vary vary VERB bracis-19024 51 9 between between ADP bracis-19024 51 10 schedulers scheduler NOUN bracis-19024 51 11 and and CCONJ bracis-19024 51 12 hpc hpc NOUN bracis-19024 51 13 facilities facility NOUN bracis-19024 51 14 , , PUNCT bracis-19024 51 15 depending depend VERB bracis-19024 51 16 on on ADP bracis-19024 51 17 whether whether SCONJ bracis-19024 51 18 the the DET bracis-19024 51 19 hosting host VERB bracis-19024 51 20 institution institution NOUN bracis-19024 51 21 prefers prefer VERB bracis-19024 51 22 to to PART bracis-19024 51 23 satisfy satisfy VERB bracis-19024 51 24 the the DET bracis-19024 51 25 needs need NOUN bracis-19024 51 26 of of ADP bracis-19024 51 27 individuals individual NOUN bracis-19024 51 28 submitting submit VERB bracis-19024 51 29 jobs job NOUN bracis-19024 51 30 , , PUNCT bracis-19024 51 31 or or CCONJ bracis-19024 51 32 the the DET bracis-19024 51 33 whole whole ADJ bracis-19024 51 34 group group NOUN bracis-19024 51 35 of of ADP bracis-19024 51 36 users user NOUN bracis-19024 51 37     SPACE bracis-19024 52 1 [ [ X bracis-19024 52 2 5 5 NUM bracis-19024 52 3 ] ] PUNCT bracis-19024 52 4 . . PUNCT bracis-19024 53 1 when when SCONJ bracis-19024 53 2 optimization optimization NOUN bracis-19024 53 3 of of ADP bracis-19024 53 4 response response NOUN bracis-19024 53 5 time time NOUN bracis-19024 53 6 is be AUX bracis-19024 53 7 a a DET bracis-19024 53 8 subgoal subgoal NOUN bracis-19024 53 9 , , PUNCT bracis-19024 53 10 it it PRON bracis-19024 53 11 is be AUX bracis-19024 53 12 usually usually ADV bracis-19024 53 13 modeled model VERB bracis-19024 53 14 as as ADP bracis-19024 53 15 the the DET bracis-19024 53 16 minimization minimization NOUN bracis-19024 53 17 of of ADP bracis-19024 53 18 the the DET bracis-19024 53 19 average average ADJ bracis-19024 53 20 response response NOUN bracis-19024 53 21 time time NOUN bracis-19024 53 22 , , PUNCT bracis-19024 53 23 with with ADP bracis-19024 53 24 response response NOUN bracis-19024 53 25 time time NOUN bracis-19024 53 26 used use VERB bracis-19024 53 27 as as ADP bracis-19024 53 28 a a DET bracis-19024 53 29 synonym synonym NOUN bracis-19024 53 30 to to PART bracis-19024 53 31 turnaround turnaround VERB bracis-19024 53 32 time time NOUN bracis-19024 53 33 : : PUNCT bracis-19024 53 34 the the DET bracis-19024 53 35 difference difference NOUN bracis-19024 53 36 between between ADP bracis-19024 53 37 the the DET bracis-19024 53 38 time time NOUN bracis-19024 53 39 a a DET bracis-19024 53 40 job job NOUN bracis-19024 53 41 was be AUX bracis-19024 53 42 submitted submit VERB bracis-19024 53 43 to to ADP bracis-19024 53 44 the the DET bracis-19024 53 45 time time NOUN bracis-19024 53 46 it it PRON bracis-19024 53 47 completed complete VERB bracis-19024 53 48 execution execution NOUN bracis-19024 53 49 . . PUNCT bracis-19024 54 1 a a DET bracis-19024 54 2 metric metric ADJ bracis-19024 54 3 commonly commonly ADV bracis-19024 54 4 used use VERB bracis-19024 54 5 to to PART bracis-19024 54 6 evaluate evaluate VERB bracis-19024 54 7 this this PRON bracis-19024 54 8 is be AUX bracis-19024 54 9 the the DET bracis-19024 54 10 slowdown slowdown NOUN bracis-19024 54 11 of of ADP bracis-19024 54 12 a a DET bracis-19024 54 13 job job NOUN bracis-19024 54 14 , , PUNCT bracis-19024 54 15 which which PRON bracis-19024 54 16 , , PUNCT bracis-19024 54 17 for for ADP bracis-19024 54 18 job job NOUN bracis-19024 54 19 j j PROPN bracis-19024 54 20 is be AUX bracis-19024 54 21 defined define VERB bracis-19024 54 22 as as ADP bracis-19024 54 23 $ $ SYM bracis-19024 54 24 $ $ SYM bracis-19024 54 25 \begin{aligned \begin{aligne VERB bracis-19024 54 26 } } PUNCT bracis-19024 54 27 \mathrm \mathrm PROPN bracis-19024 54 28 { { PUNCT bracis-19024 54 29 slowdown}(j)=\frac{(t_f(j)-t_s(j))}{t_e(j slowdown}(j)=\frac{(t_f(j)-t_s(j))}{t_e(j PROPN bracis-19024 54 30 ) ) PUNCT bracis-19024 54 31 } } PUNCT bracis-19024 54 32 = = SYM bracis-19024 54 33 \frac{t_w(j)+t_e(j)}{t_e(j \frac{t_w(j)+t_e(j)}{t_e(j NOUN bracis-19024 54 34 ) ) PUNCT bracis-19024 54 35 } } PUNCT bracis-19024 54 36 = = PUNCT bracis-19024 54 37 \frac{1}{t_e(j)}\left \frac{1}{t_e(j)}\left PROPN bracis-19024 54 38 ( ( PUNCT bracis-19024 54 39 \sum \sum NOUN bracis-19024 54 40 _ _ PUNCT bracis-19024 54 41 { { PUNCT bracis-19024 55 1 i=1}^{t_w(j)}1+\sum i=1}^{t_w(j)}1+\sum NUM bracis-19024 55 2 _ _ NOUN bracis-19024 55 3 { { PUNCT bracis-19024 55 4 i=1}^{t_e(j)}1\right i=1}^{t_e(j)}1\right VERB bracis-19024 55 5 ) ) PUNCT bracis-19024 55 6 \text \text NOUN bracis-19024 55 7 { { PUNCT bracis-19024 55 8 , , PUNCT bracis-19024 55 9 } } PUNCT bracis-19024 55 10 \end{aligned}$$ \end{aligned}$$ X bracis-19024 55 11 ( ( PUNCT bracis-19024 55 12 1 1 NUM bracis-19024 55 13 ) ) PUNCT bracis-19024 56 1 where where SCONJ bracis-19024 56 2 \(t_s(j)\ \(t_s(j)\ ADJ bracis-19024 56 3 ) ) PUNCT bracis-19024 56 4 is be AUX bracis-19024 56 5 the the DET bracis-19024 56 6 time time NOUN bracis-19024 56 7 job job NOUN bracis-19024 56 8 j j PROPN bracis-19024 56 9 was be AUX bracis-19024 56 10 submitted submit VERB bracis-19024 56 11 , , PUNCT bracis-19024 56 12 \(t_e(j)\ \(t_e(j)\ X bracis-19024 56 13 ) ) PUNCT bracis-19024 56 14 is be AUX bracis-19024 56 15 the the DET bracis-19024 56 16 time time NOUN bracis-19024 56 17 it it PRON bracis-19024 56 18 took take VERB bracis-19024 56 19 to to PART bracis-19024 56 20 execute execute VERB bracis-19024 56 21 job job NOUN bracis-19024 56 22 j j PROPN bracis-19024 56 23 , , PUNCT bracis-19024 56 24 and and CCONJ bracis-19024 56 25 \(t_f(j)\ \(t_f(j)\ NOUN bracis-19024 56 26 ) ) PUNCT bracis-19024 56 27 is be AUX bracis-19024 56 28 the the DET bracis-19024 56 29 finish finish NOUN bracis-19024 56 30 time time NOUN bracis-19024 56 31 of of ADP bracis-19024 56 32 job job NOUN bracis-19024 56 33 j. j. PROPN bracis-19024 56 34 the the DET bracis-19024 56 35 equality equality NOUN bracis-19024 56 36 in in ADP bracis-19024 56 37 the the DET bracis-19024 56 38 middle middle NOUN bracis-19024 56 39 holds hold NOUN bracis-19024 56 40 because because SCONJ bracis-19024 56 41 the the DET bracis-19024 56 42 wait wait NOUN bracis-19024 56 43 time time NOUN bracis-19024 56 44 , , PUNCT bracis-19024 56 45 \(t_w\ \(t_w\ NOUN bracis-19024 56 46 ) ) PUNCT bracis-19024 56 47 , , PUNCT bracis-19024 56 48 of of ADP bracis-19024 56 49 a a DET bracis-19024 56 50 job job NOUN bracis-19024 56 51 j j NOUN bracis-19024 56 52 is be AUX bracis-19024 56 53 defined define VERB bracis-19024 56 54 as as ADP bracis-19024 56 55 \(t_w(j)=t_f(j)-(t_e(j)\,+\,t_s(j))\ \(t_w(j)=t_f(j)-(t_e(j)\,+\,t_s(j))\ NOUN bracis-19024 56 56 ) ) PUNCT bracis-19024 56 57 . . PUNCT bracis-19024 57 1 fig fig NOUN bracis-19024 57 2 . . PUNCT bracis-19024 58 1 1 1 X bracis-19024 58 2 . . X bracis-19024 58 3 a a DET bracis-19024 58 4 possible possible ADJ bracis-19024 58 5 schedule schedule NOUN bracis-19024 58 6 when when SCONJ bracis-19024 58 7 three three NUM bracis-19024 58 8 jobs job NOUN bracis-19024 58 9 arrive arrive VERB bracis-19024 58 10 in in ADP bracis-19024 58 11 a a DET bracis-19024 58 12 scheduling scheduling NOUN bracis-19024 58 13 system system NOUN bracis-19024 58 14 at at ADP bracis-19024 58 15 discrete discrete ADJ bracis-19024 58 16 time time NOUN bracis-19024 58 17 step step NOUN bracis-19024 58 18 1 1 NUM bracis-19024 58 19 and and CCONJ bracis-19024 58 20 no no DET bracis-19024 58 21 more more ADJ bracis-19024 58 22 jobs job NOUN bracis-19024 58 23 are be AUX bracis-19024 58 24 submitted submit VERB bracis-19024 58 25 to to ADP bracis-19024 58 26 the the DET bracis-19024 58 27 system system NOUN bracis-19024 58 28 at at ADP bracis-19024 58 29 least least ADJ bracis-19024 58 30 until until ADP bracis-19024 58 31 time time NOUN bracis-19024 58 32 step step NOUN bracis-19024 58 33 7 7 NUM bracis-19024 58 34 , , PUNCT bracis-19024 58 35 the the DET bracis-19024 58 36 last last ADJ bracis-19024 58 37 one one NOUN bracis-19024 58 38 shown show VERB bracis-19024 58 39 in in ADP bracis-19024 58 40 the the DET bracis-19024 58 41 figure figure NOUN bracis-19024 58 42 . . PUNCT bracis-19024 59 1 full full ADJ bracis-19024 59 2 size size NOUN bracis-19024 59 3 image image NOUN bracis-19024 59 4 consider consider VERB bracis-19024 59 5 the the DET bracis-19024 59 6 case case NOUN bracis-19024 59 7 of of ADP bracis-19024 59 8 three three NUM bracis-19024 59 9 batch batch NOUN bracis-19024 59 10 jobs job NOUN bracis-19024 59 11 , , PUNCT bracis-19024 59 12 \(j_1=\ \(j_1=\ NUM bracis-19024 59 13 ) ) PUNCT bracis-19024 59 14 , , PUNCT bracis-19024 59 15 \(j_2=\ \(j_2=\ NUM bracis-19024 59 16 ) ) PUNCT bracis-19024 59 17 , , PUNCT bracis-19024 59 18 and and CCONJ bracis-19024 59 19 \(j_3=\ \(j_3=\ NOUN bracis-19024 59 20 ) ) PUNCT bracis-19024 59 21 , , PUNCT bracis-19024 59 22 submitted submit VERB bracis-19024 59 23 to to ADP bracis-19024 59 24 a a DET bracis-19024 59 25 scheduling scheduling NOUN bracis-19024 59 26 system system NOUN bracis-19024 59 27 with with ADP bracis-19024 59 28 two two NUM bracis-19024 59 29 processors processor NOUN bracis-19024 59 30 , , PUNCT bracis-19024 59 31 and and CCONJ bracis-19024 59 32 that that SCONJ bracis-19024 59 33 the the DET bracis-19024 59 34 three three NUM bracis-19024 59 35 jobs job NOUN bracis-19024 59 36 were be AUX bracis-19024 59 37 submitted submit VERB bracis-19024 59 38 “ " PUNCT bracis-19024 59 39 between between ADP bracis-19024 59 40 ” " PUNCT bracis-19024 59 41 time time NOUN bracis-19024 59 42 step step NOUN bracis-19024 59 43 0 0 NUM bracis-19024 59 44 and and CCONJ bracis-19024 59 45 1 1 NUM bracis-19024 59 46 , , PUNCT bracis-19024 59 47 such such ADJ bracis-19024 59 48 that that SCONJ bracis-19024 59 49 , , PUNCT bracis-19024 59 50 when when SCONJ bracis-19024 59 51 transitioning transition VERB bracis-19024 59 52 from from ADP bracis-19024 59 53 the the DET bracis-19024 59 54 first first ADJ bracis-19024 59 55 time time NOUN bracis-19024 59 56 step step NOUN bracis-19024 59 57 to to ADP bracis-19024 59 58 the the DET bracis-19024 59 59 second second ADJ bracis-19024 59 60 , , PUNCT bracis-19024 59 61 now now ADV bracis-19024 59 62 there there PRON bracis-19024 59 63 are be VERB bracis-19024 59 64 three three NUM bracis-19024 59 65 jobs job NOUN bracis-19024 59 66 waiting wait VERB bracis-19024 59 67 . . PUNCT bracis-19024 60 1 also also ADV bracis-19024 60 2 consider consider VERB bracis-19024 60 3 that that PRON bracis-19024 60 4 , , PUNCT bracis-19024 60 5 for for ADP bracis-19024 60 6 these these DET bracis-19024 60 7 jobs job NOUN bracis-19024 60 8 , , PUNCT bracis-19024 60 9 the the DET bracis-19024 60 10 generated generate VERB bracis-19024 60 11 schedule schedule NOUN bracis-19024 60 12 is be AUX bracis-19024 60 13 the the DET bracis-19024 60 14 one one NUM bracis-19024 60 15 displayed display VERB bracis-19024 60 16 in in ADP bracis-19024 60 17 fig fig NOUN bracis-19024 60 18 . . PUNCT bracis-19024 60 19     SPACE bracis-19024 61 1 1 1 NUM bracis-19024 61 2 . . PUNCT bracis-19024 61 3 as as SCONJ bracis-19024 61 4 shown show VERB bracis-19024 61 5 in in ADP bracis-19024 61 6 the the DET bracis-19024 61 7 figure figure NOUN bracis-19024 61 8 , , PUNCT bracis-19024 61 9 the the DET bracis-19024 61 10 jobs job NOUN bracis-19024 61 11 execute execute VERB bracis-19024 61 12 for for ADP bracis-19024 61 13 two two NUM bracis-19024 61 14 , , PUNCT bracis-19024 61 15 three three NUM bracis-19024 61 16 and and CCONJ bracis-19024 61 17 four four NUM bracis-19024 61 18 time time NOUN bracis-19024 61 19 steps step NOUN bracis-19024 61 20 respectively respectively ADV bracis-19024 61 21 , , PUNCT bracis-19024 61 22 and and CCONJ bracis-19024 61 23 all all PRON bracis-19024 61 24 of of ADP bracis-19024 61 25 them they PRON bracis-19024 61 26 use use VERB bracis-19024 61 27 a a DET bracis-19024 61 28 single single ADJ bracis-19024 61 29 processor processor NOUN bracis-19024 61 30 . . PUNCT bracis-19024 62 1 the the DET bracis-19024 62 2 reader reader NOUN bracis-19024 62 3 should should AUX bracis-19024 62 4 observe observe VERB bracis-19024 62 5 that that SCONJ bracis-19024 62 6 different different ADJ bracis-19024 62 7 schedules schedule NOUN bracis-19024 62 8 can can AUX bracis-19024 62 9 yield yield VERB bracis-19024 62 10 substantially substantially ADV bracis-19024 62 11 different different ADJ bracis-19024 62 12 values value NOUN bracis-19024 62 13 of of ADP bracis-19024 62 14 ( ( PUNCT bracis-19024 62 15 average average ADJ bracis-19024 62 16 ) ) PUNCT bracis-19024 62 17 slowdown slowdown NOUN bracis-19024 62 18 . . PUNCT bracis-19024 63 1 for for ADP bracis-19024 63 2 example example NOUN bracis-19024 63 3 , , PUNCT bracis-19024 63 4 the the DET bracis-19024 63 5 schedule schedule NOUN bracis-19024 63 6 shown show VERB bracis-19024 63 7 in in ADP bracis-19024 63 8 fig fig NOUN bracis-19024 63 9 . . PUNCT bracis-19024 63 10     SPACE bracis-19024 63 11 1 1 NUM bracis-19024 63 12 has have VERB bracis-19024 63 13 an an DET bracis-19024 63 14 average average ADJ bracis-19024 63 15 slowdown slowdown NOUN bracis-19024 63 16 equal equal ADJ bracis-19024 63 17 to to ADP bracis-19024 63 18 , , PUNCT bracis-19024 63 19 whereas whereas SCONJ bracis-19024 63 20 , , PUNCT bracis-19024 63 21 if if SCONJ bracis-19024 63 22 we we PRON bracis-19024 63 23 swapped swap VERB bracis-19024 63 24 \(j_3\ \(j_3\ NOUN bracis-19024 63 25 ) ) PUNCT bracis-19024 63 26 with with ADP bracis-19024 63 27 \(j_1\ \(j_1\ PROPN bracis-19024 63 28 ) ) PUNCT bracis-19024 63 29 , , PUNCT bracis-19024 63 30 and and CCONJ bracis-19024 63 31 started start VERB bracis-19024 63 32 \(j_1\ \(j_1\ PROPN bracis-19024 63 33 ) ) PUNCT bracis-19024 63 34 soon soon ADV bracis-19024 63 35 after after SCONJ bracis-19024 63 36 \(j_2\ \(j_2\ PROPN bracis-19024 63 37 ) ) PUNCT bracis-19024 63 38 finished finish VERB bracis-19024 63 39 , , PUNCT bracis-19024 63 40 the the DET bracis-19024 63 41 slowdown slowdown NOUN bracis-19024 63 42 would would AUX bracis-19024 63 43 be be AUX bracis-19024 63 44 , , PUNCT bracis-19024 63 45 a a DET bracis-19024 63 46 \(\approx \(\approx NOUN bracis-19024 63 47 29\%\ 29\%\ ADJ bracis-19024 63 48 ) ) PUNCT bracis-19024 63 49 increase increase NOUN bracis-19024 63 50 . . PUNCT bracis-19024 64 1 therefore therefore ADV bracis-19024 64 2 , , PUNCT bracis-19024 64 3 a a DET bracis-19024 64 4 scheduler scheduler NOUN bracis-19024 64 5 should should AUX bracis-19024 64 6 choose choose VERB bracis-19024 64 7 job job NOUN bracis-19024 64 8 sequences sequence NOUN bracis-19024 64 9 wisely wisely ADV bracis-19024 64 10 , , PUNCT bracis-19024 64 11 otherwise otherwise ADV bracis-19024 64 12 its its PRON bracis-19024 64 13 performance performance NOUN bracis-19024 64 14 can can AUX bracis-19024 64 15 be be AUX bracis-19024 64 16 degraded degrade VERB bracis-19024 64 17 . . PUNCT bracis-19024 65 1 in in ADP bracis-19024 65 2 this this DET bracis-19024 65 3 paper paper NOUN bracis-19024 65 4 , , PUNCT bracis-19024 65 5 we we PRON bracis-19024 65 6 will will AUX bracis-19024 65 7 focus focus VERB bracis-19024 65 8 our our PRON bracis-19024 65 9 discussion discussion NOUN bracis-19024 65 10 on on ADP bracis-19024 65 11 what what PRON bracis-19024 65 12 happens happen VERB bracis-19024 65 13 when when SCONJ bracis-19024 65 14 an an DET bracis-19024 65 15 rl rl NOUN bracis-19024 65 16 system system NOUN bracis-19024 65 17 tries try VERB bracis-19024 65 18 to to PART bracis-19024 65 19 minimize minimize VERB bracis-19024 65 20 the the DET bracis-19024 65 21 average average ADJ bracis-19024 65 22 slowdown slowdown NOUN bracis-19024 65 23 , , PUNCT bracis-19024 65 24 but but CCONJ bracis-19024 65 25 our our PRON bracis-19024 65 26 conclusions conclusion NOUN bracis-19024 65 27 are be AUX bracis-19024 65 28 general general ADJ bracis-19024 65 29 and and CCONJ bracis-19024 65 30 apply apply VERB bracis-19024 65 31 to to ADP bracis-19024 65 32 other other ADJ bracis-19024 65 33 metrics metric NOUN bracis-19024 65 34 and and CCONJ bracis-19024 65 35 problems problem NOUN bracis-19024 65 36 as as ADV bracis-19024 65 37 well well ADV bracis-19024 65 38 . . PUNCT bracis-19024 66 1 3.2 3.2 NUM bracis-19024 66 2 deep deep ADJ bracis-19024 66 3 reinforcement reinforcement NOUN bracis-19024 66 4 learning learning NOUN bracis-19024 66 5 and and CCONJ bracis-19024 66 6 job job NOUN bracis-19024 66 7 scheduling scheduling NOUN bracis-19024 66 8 in in ADP bracis-19024 66 9 a a DET bracis-19024 66 10 reinforcement reinforcement NOUN bracis-19024 66 11 learning learning NOUN bracis-19024 66 12 ( ( PUNCT bracis-19024 66 13 rl rl NOUN bracis-19024 66 14 ) ) PUNCT bracis-19024 66 15 problem problem NOUN bracis-19024 66 16 , , PUNCT bracis-19024 66 17 an an DET bracis-19024 66 18 agent agent NOUN bracis-19024 66 19 interacts interact VERB bracis-19024 66 20 with with ADP bracis-19024 66 21 an an DET bracis-19024 66 22 unknown unknown ADJ bracis-19024 66 23 environment environment NOUN bracis-19024 66 24 in in ADP bracis-19024 66 25 which which PRON bracis-19024 66 26 it it PRON bracis-19024 66 27 attempts attempt VERB bracis-19024 66 28 to to PART bracis-19024 66 29 optimize optimize VERB bracis-19024 66 30 a a DET bracis-19024 66 31 reward reward NOUN bracis-19024 66 32 signal signal NOUN bracis-19024 66 33 by by ADP bracis-19024 66 34 sequentially sequentially ADV bracis-19024 66 35 observing observe VERB bracis-19024 66 36 the the DET bracis-19024 66 37 environment environment NOUN bracis-19024 66 38 ’s ’s PART bracis-19024 66 39 state state NOUN bracis-19024 66 40 and and CCONJ bracis-19024 66 41 taking take VERB bracis-19024 66 42 actions action NOUN bracis-19024 66 43 according accord VERB bracis-19024 66 44 to to ADP bracis-19024 66 45 its its PRON bracis-19024 66 46 perception perception NOUN bracis-19024 66 47 . . PUNCT bracis-19024 67 1 for for ADP bracis-19024 67 2 each each DET bracis-19024 67 3 action action NOUN bracis-19024 67 4 , , PUNCT bracis-19024 67 5 the the DET bracis-19024 67 6 agent agent NOUN bracis-19024 67 7 receives receive VERB bracis-19024 67 8 a a DET bracis-19024 67 9 reward reward NOUN bracis-19024 67 10 . . PUNCT bracis-19024 68 1 thus thus ADV bracis-19024 68 2 , , PUNCT bracis-19024 68 3 in in ADP bracis-19024 68 4 the the DET bracis-19024 68 5 end end NOUN bracis-19024 68 6 , , PUNCT bracis-19024 68 7 we we PRON bracis-19024 68 8 want want VERB bracis-19024 68 9 to to PART bracis-19024 68 10 find find VERB bracis-19024 68 11 the the DET bracis-19024 68 12 sequence sequence NOUN bracis-19024 68 13 of of ADP bracis-19024 68 14 actions action NOUN bracis-19024 68 15 that that PRON bracis-19024 68 16 maximizes maximize VERB bracis-19024 68 17 the the DET bracis-19024 68 18 total total ADJ bracis-19024 68 19 reward reward NOUN bracis-19024 68 20 , , PUNCT bracis-19024 68 21 as as SCONJ bracis-19024 68 22 we we PRON bracis-19024 68 23 will will AUX bracis-19024 68 24 detail detail VERB bracis-19024 68 25 in in ADP bracis-19024 68 26 the the DET bracis-19024 68 27 next next ADJ bracis-19024 68 28 paragraphs paragraph NOUN bracis-19024 68 29 . . PUNCT bracis-19024 69 1 rl rl PROPN bracis-19024 69 2 formalizes formalize VERB bracis-19024 69 3 the the DET bracis-19024 69 4 problem problem NOUN bracis-19024 69 5 as as ADP bracis-19024 69 6 a a DET bracis-19024 69 7 markov markov NOUN bracis-19024 69 8 decision decision NOUN bracis-19024 69 9 process process NOUN bracis-19024 69 10 ( ( PUNCT bracis-19024 69 11 mdp mdp NOUN bracis-19024 69 12 ) ) PUNCT bracis-19024 69 13 represented represent VERB bracis-19024 69 14 by by ADP bracis-19024 69 15 a a DET bracis-19024 69 16 tuple tuple NOUN bracis-19024 69 17 \(\mathcal \(\mathcal ADJ bracis-19024 69 18 { { PUNCT bracis-19024 69 19 m m NOUN bracis-19024 69 20 } } PUNCT bracis-19024 69 21 = = PUNCT bracis-19024 69 22 \langle \langle PROPN bracis-19024 69 23 \mathcal \mathcal PROPN bracis-19024 69 24 { { PUNCT bracis-19024 69 25 s s PROPN bracis-19024 69 26 } } PUNCT bracis-19024 69 27 , , PUNCT bracis-19024 69 28 \mathcal \mathcal PROPN bracis-19024 69 29 { { PUNCT bracis-19024 69 30 a a NOUN bracis-19024 69 31 } } PUNCT bracis-19024 69 32 , , PUNCT bracis-19024 69 33 \mathcal \mathcal X bracis-19024 69 34 { { PUNCT bracis-19024 69 35 r r NOUN bracis-19024 69 36 } } PUNCT bracis-19024 69 37 , , PUNCT bracis-19024 69 38 \mathcal \mathcal PROPN bracis-19024 69 39 { { PUNCT bracis-19024 69 40 t t NOUN bracis-19024 69 41 } } PUNCT bracis-19024 69 42 , , PUNCT bracis-19024 69 43 \rho \rho PROPN bracis-19024 69 44 , , PUNCT bracis-19024 69 45 \gamma \gamma PROPN bracis-19024 69 46 \rangle \rangle PROPN bracis-19024 69 47 \)footnote \)footnote NUM bracis-19024 69 48 2 2 NUM bracis-19024 69 49 . . PUNCT bracis-19024 70 1 at at ADP bracis-19024 70 2 each each DET bracis-19024 70 3 discrete discrete ADJ bracis-19024 70 4 time time NOUN bracis-19024 70 5 step step NOUN bracis-19024 70 6 t t PROPN bracis-19024 70 7 the the DET bracis-19024 70 8 agent agent NOUN bracis-19024 70 9 is be AUX bracis-19024 70 10 in in ADP bracis-19024 70 11 state state NOUN bracis-19024 70 12 \(s_t \(s_t PROPN bracis-19024 70 13 \in \in PROPN bracis-19024 70 14 \mathcal \mathcal PROPN bracis-19024 70 15 { { PUNCT bracis-19024 70 16 s}\ s}\ NUM bracis-19024 70 17 ) ) PUNCT bracis-19024 70 18 . . PUNCT bracis-19024 71 1 from from ADP bracis-19024 71 2 \(s_t\ \(s_t\ ADJ bracis-19024 71 3 ) ) PUNCT bracis-19024 71 4 , , PUNCT bracis-19024 71 5 the the DET bracis-19024 71 6 agent agent NOUN bracis-19024 71 7 takes take VERB bracis-19024 71 8 an an DET bracis-19024 71 9 action action NOUN bracis-19024 71 10 \(a_t \(a_t ADV bracis-19024 71 11 \in \in PROPN bracis-19024 71 12 \mathcal \mathcal PROPN bracis-19024 71 13 { { PUNCT bracis-19024 71 14 a}\ a}\ NUM bracis-19024 71 15 ) ) PUNCT bracis-19024 71 16 , , PUNCT bracis-19024 71 17 receives receive VERB bracis-19024 71 18 reward reward NOUN bracis-19024 71 19 \(r_{t+1 \(r_{t+1 NOUN bracis-19024 71 20 } } PUNCT bracis-19024 71 21 \in \in PROPN bracis-19024 71 22 \mathcal \mathcal PROPN bracis-19024 71 23 { { PUNCT bracis-19024 71 24 r}\ r}\ NUM bracis-19024 71 25 ) ) PUNCT bracis-19024 71 26 and and CCONJ bracis-19024 71 27 ends end VERB bracis-19024 71 28 up up ADP bracis-19024 71 29 in in ADP bracis-19024 71 30 state state PROPN bracis-19024 71 31 \(s_{t+1 \(s_{t+1 PROPN bracis-19024 71 32 } } PUNCT bracis-19024 71 33 \in \in PROPN bracis-19024 71 34 \mathcal \mathcal PROPN bracis-19024 71 35 { { PUNCT bracis-19024 71 36 s}\ s}\ NUM bracis-19024 71 37 ) ) PUNCT bracis-19024 71 38 . . PUNCT bracis-19024 72 1 therefore therefore ADV bracis-19024 72 2 , , PUNCT bracis-19024 72 3 when when SCONJ bracis-19024 72 4 we we PRON bracis-19024 72 5 assume assume VERB bracis-19024 72 6 the the DET bracis-19024 72 7 first first ADJ bracis-19024 72 8 time time NOUN bracis-19024 72 9 step step NOUN bracis-19024 72 10 is be AUX bracis-19024 72 11 0 0 NUM bracis-19024 72 12 , , PUNCT bracis-19024 72 13 the the DET bracis-19024 72 14 interaction interaction NOUN bracis-19024 72 15 between between ADP bracis-19024 72 16 agent agent NOUN bracis-19024 72 17 and and CCONJ bracis-19024 72 18 environment environment NOUN bracis-19024 72 19 create create VERB bracis-19024 72 20 a a DET bracis-19024 72 21 sequence sequence NOUN bracis-19024 72 22 \(s_0 \(s_0 PROPN bracis-19024 72 23 , , PUNCT bracis-19024 72 24 a_0 a_0 NOUN bracis-19024 72 25 , , PUNCT bracis-19024 72 26 r_1 r_1 NOUN bracis-19024 72 27 , , PUNCT bracis-19024 72 28 s_1 s_1 NOUN bracis-19024 72 29 , , PUNCT bracis-19024 72 30 a_1 a_1 PROPN bracis-19024 72 31 , , PUNCT bracis-19024 72 32 r_2 r_2 PROPN bracis-19024 72 33 , , PUNCT bracis-19024 72 34 \ldots \ldots PROPN bracis-19024 72 35 \ \ NOUN bracis-19024 72 36 ) ) PUNCT bracis-19024 72 37 of of ADP bracis-19024 72 38 states state NOUN bracis-19024 72 39 , , PUNCT bracis-19024 72 40 actions action NOUN bracis-19024 72 41 and and CCONJ bracis-19024 72 42 rewards reward NOUN bracis-19024 72 43 . . PUNCT bracis-19024 73 1 to to ADP bracis-19024 73 2 a a DET bracis-19024 73 3 specific specific ADJ bracis-19024 73 4 sequence sequence NOUN bracis-19024 73 5 \(s_0 \(s_0 PROPN bracis-19024 73 6 , , PUNCT bracis-19024 73 7 a_0 a_0 NOUN bracis-19024 73 8 , , PUNCT bracis-19024 73 9 r_1 r_1 NOUN bracis-19024 73 10 , , PUNCT bracis-19024 73 11 s_1 s_1 NOUN bracis-19024 73 12 , , PUNCT bracis-19024 73 13 a_1 a_1 PROPN bracis-19024 73 14 , , PUNCT bracis-19024 73 15 r_2 r_2 PROPN bracis-19024 73 16 , , PUNCT bracis-19024 73 17 \ldots \ldots PROPN bracis-19024 73 18 \ \ NOUN bracis-19024 73 19 ) ) PUNCT bracis-19024 73 20 of of ADP bracis-19024 73 21 states state NOUN bracis-19024 73 22 , , PUNCT bracis-19024 73 23 actions action NOUN bracis-19024 73 24 , , PUNCT bracis-19024 73 25 and and CCONJ bracis-19024 73 26 rewards reward NOUN bracis-19024 73 27 we we PRON bracis-19024 73 28 give give VERB bracis-19024 73 29 the the DET bracis-19024 73 30 name name NOUN bracis-19024 73 31 of of ADP bracis-19024 73 32 trajectory trajectory NOUN bracis-19024 73 33 , , PUNCT bracis-19024 73 34 and and CCONJ bracis-19024 73 35 will will AUX bracis-19024 73 36 denote denote VERB bracis-19024 73 37 such such ADJ bracis-19024 73 38 sequences sequence NOUN bracis-19024 73 39 by by ADP bracis-19024 73 40 \(\tau \(\tau NOUN bracis-19024 73 41 \ \ NUM bracis-19024 73 42 ) ) PUNCT bracis-19024 73 43 . . PUNCT bracis-19024 74 1 the the DET bracis-19024 74 2 transition transition NOUN bracis-19024 74 3 from from ADP bracis-19024 74 4 state state PROPN bracis-19024 74 5 \(s_t\ \(s_t\ NOUN bracis-19024 74 6 ) ) PUNCT bracis-19024 74 7 to to ADP bracis-19024 74 8 \(s_{t+1}\ \(s_{t+1}\ NUM bracis-19024 74 9 ) ) PUNCT bracis-19024 74 10 follows follow VERB bracis-19024 74 11 the the DET bracis-19024 74 12 probability probability NOUN bracis-19024 74 13 distribution distribution NOUN bracis-19024 74 14 defined define VERB bracis-19024 74 15 by by ADP bracis-19024 74 16 \(\mathcal \(\mathcal ADJ bracis-19024 74 17 { { PUNCT bracis-19024 74 18 t t NOUN bracis-19024 74 19 } } PUNCT bracis-19024 74 20 : : PUNCT bracis-19024 74 21 \mathcal \mathcal PROPN bracis-19024 74 22 { { PUNCT bracis-19024 74 23 s s NOUN bracis-19024 74 24 } } PUNCT bracis-19024 74 25 \times \times ADP bracis-19024 74 26 \mathcal \mathcal PROPN bracis-19024 74 27 { { PUNCT bracis-19024 74 28 a a NOUN bracis-19024 74 29 } } PUNCT bracis-19024 74 30 \rightarrow \rightarrow PROPN bracis-19024 74 31 \mathcal \mathcal PROPN bracis-19024 74 32 { { PUNCT bracis-19024 74 33 s}\ s}\ PROPN bracis-19024 74 34 ) ) PUNCT bracis-19024 74 35 or or CCONJ bracis-19024 74 36 , , PUNCT bracis-19024 74 37 in in ADP bracis-19024 74 38 an an DET bracis-19024 74 39 equivalent equivalent ADJ bracis-19024 74 40 way way NOUN bracis-19024 74 41 , , PUNCT bracis-19024 74 42 \(\mathcal \(\mathcal ADJ bracis-19024 74 43 { { PUNCT bracis-19024 74 44 t}\ t}\ NUM bracis-19024 74 45 ) ) PUNCT bracis-19024 74 46 gives give VERB bracis-19024 74 47 the the DET bracis-19024 74 48 probability probability NOUN bracis-19024 74 49 of of ADP bracis-19024 74 50 reaching reach VERB bracis-19024 74 51 any any DET bracis-19024 74 52 new new ADJ bracis-19024 74 53 state state NOUN bracis-19024 74 54 \(s'\ \(s'\ PROPN bracis-19024 74 55 ) ) PUNCT bracis-19024 74 56 when when SCONJ bracis-19024 74 57 taking take VERB bracis-19024 74 58 action action NOUN bracis-19024 74 59 a a PRON bracis-19024 74 60 when when SCONJ bracis-19024 74 61 in in ADP bracis-19024 74 62 state state NOUN bracis-19024 74 63 s s NOUN bracis-19024 74 64 : : PUNCT bracis-19024 74 65 \(p(s'|s \(p(s'|s VERB bracis-19024 74 66 , , PUNCT bracis-19024 74 67 a a PRON bracis-19024 74 68 ) ) PUNCT bracis-19024 74 69 = = PUNCT bracis-19024 74 70 p(s_{t+1}=s p(s_{t+1}=s NOUN bracis-19024 74 71 ' ' PUNCT bracis-19024 74 72 | | ADV bracis-19024 74 73 s_t s_t NOUN bracis-19024 74 74 = = NOUN bracis-19024 74 75 s s PART bracis-19024 74 76 , , PUNCT bracis-19024 74 77 a_t a_t PUNCT bracis-19024 74 78 = = NOUN bracis-19024 74 79 a)\ a)\ NUM bracis-19024 74 80 ) ) PUNCT bracis-19024 74 81 . . PUNCT bracis-19024 75 1 \(\rho \(\rho PROPN bracis-19024 76 1 \ \ NOUN bracis-19024 76 2 ) ) PUNCT bracis-19024 76 3 is be AUX bracis-19024 76 4 a a DET bracis-19024 76 5 distribution distribution NOUN bracis-19024 76 6 of of ADP bracis-19024 76 7 initial initial ADJ bracis-19024 76 8 states state NOUN bracis-19024 76 9 , , PUNCT bracis-19024 76 10 and and CCONJ bracis-19024 76 11 \(\gamma \(\gamma X bracis-19024 76 12 \ \ NOUN bracis-19024 76 13 ) ) PUNCT bracis-19024 76 14 is be AUX bracis-19024 76 15 a a DET bracis-19024 76 16 parameter parameter NOUN bracis-19024 76 17 \(0 \(0 PROPN bracis-19024 76 18 \le \le PROPN bracis-19024 76 19 \gamma \gamma PROPN bracis-19024 76 20 \le \le PROPN bracis-19024 76 21 1\ 1\ NUM bracis-19024 76 22 ) ) PUNCT bracis-19024 76 23 , , PUNCT bracis-19024 76 24 called call VERB bracis-19024 76 25 the the DET bracis-19024 76 26 discount discount NOUN bracis-19024 76 27 rate rate NOUN bracis-19024 76 28 . . PUNCT bracis-19024 77 1 the the DET bracis-19024 77 2 discount discount NOUN bracis-19024 77 3 rate rate NOUN bracis-19024 77 4 models model NOUN bracis-19024 77 5 the the DET bracis-19024 77 6 present present ADJ bracis-19024 77 7 value value NOUN bracis-19024 77 8 of of ADP bracis-19024 77 9 future future ADJ bracis-19024 77 10 rewards reward NOUN bracis-19024 77 11 . . PUNCT bracis-19024 78 1 for for ADP bracis-19024 78 2 example example NOUN bracis-19024 78 3 , , PUNCT bracis-19024 78 4 a a DET bracis-19024 78 5 reward reward NOUN bracis-19024 78 6 received receive VERB bracis-19024 78 7 k k PROPN bracis-19024 78 8 steps step NOUN bracis-19024 78 9 in in ADP bracis-19024 78 10 the the DET bracis-19024 78 11 future future NOUN bracis-19024 78 12 is be AUX bracis-19024 78 13 worth worth ADJ bracis-19024 78 14 only only ADV bracis-19024 78 15 \(\gamma \(\gamma ADP bracis-19024 78 16 ^k\ ^k\ NOUN bracis-19024 78 17 ) ) PUNCT bracis-19024 78 18 now now ADV bracis-19024 78 19 . . PUNCT bracis-19024 79 1 this this DET bracis-19024 79 2 discount discount NOUN bracis-19024 79 3 factor factor NOUN bracis-19024 79 4 is be AUX bracis-19024 79 5 added add VERB bracis-19024 79 6 due due ADP bracis-19024 79 7 to to ADP bracis-19024 79 8 the the DET bracis-19024 79 9 uncertainty uncertainty NOUN bracis-19024 79 10 in in ADP bracis-19024 79 11 receiving receive VERB bracis-19024 79 12 rewards reward NOUN bracis-19024 79 13 and and CCONJ bracis-19024 79 14 is be AUX bracis-19024 79 15 useful useful ADJ bracis-19024 79 16 for for ADP bracis-19024 79 17 modeling model VERB bracis-19024 79 18 stochastic stochastic ADJ bracis-19024 79 19 environments environment NOUN bracis-19024 79 20 . . PUNCT bracis-19024 80 1 in in ADP bracis-19024 80 2 such such ADJ bracis-19024 80 3 cases case NOUN bracis-19024 80 4 , , PUNCT bracis-19024 80 5 there there PRON bracis-19024 80 6 is be VERB bracis-19024 80 7 no no DET bracis-19024 80 8 guarantee guarantee NOUN bracis-19024 80 9 an an DET bracis-19024 80 10 anticipated anticipated ADJ bracis-19024 80 11 reward reward NOUN bracis-19024 80 12 will will AUX bracis-19024 80 13 actually actually ADV bracis-19024 80 14 be be AUX bracis-19024 80 15 received receive VERB bracis-19024 80 16 and and CCONJ bracis-19024 80 17 the the DET bracis-19024 80 18 discount discount NOUN bracis-19024 80 19 rate rate NOUN bracis-19024 80 20 models model NOUN bracis-19024 80 21 this this DET bracis-19024 80 22 uncertainty uncertainty NOUN bracis-19024 80 23 . . PUNCT bracis-19024 81 1 to to PART bracis-19024 81 2 map map VERB bracis-19024 81 3 our our PRON bracis-19024 81 4 presentation presentation NOUN bracis-19024 81 5 of of ADP bracis-19024 81 6 rl rl NOUN bracis-19024 81 7 into into ADP bracis-19024 81 8 our our PRON bracis-19024 81 9 problem problem NOUN bracis-19024 81 10 of of ADP bracis-19024 81 11 job job NOUN bracis-19024 81 12 scheduling scheduling NOUN bracis-19024 81 13 , , PUNCT bracis-19024 81 14 we we PRON bracis-19024 81 15 consider consider VERB bracis-19024 81 16 ( ( PUNCT bracis-19024 81 17 the the DET bracis-19024 81 18 only only ADJ bracis-19024 81 19 possible possible ADJ bracis-19024 81 20 initial initial ADJ bracis-19024 81 21 state state NOUN bracis-19024 81 22 is be AUX bracis-19024 81 23 the the DET bracis-19024 81 24 empty empty ADJ bracis-19024 81 25 cluster cluster NOUN bracis-19024 81 26 ) ) PUNCT bracis-19024 81 27 , , PUNCT bracis-19024 81 28 with with ADP bracis-19024 81 29 the the DET bracis-19024 81 30 first first ADJ bracis-19024 81 31 state state NOUN bracis-19024 81 32 consisting consist VERB bracis-19024 81 33 of of ADP bracis-19024 81 34 the the DET bracis-19024 81 35 empty empty ADJ bracis-19024 81 36 cluster cluster NOUN bracis-19024 81 37 , , PUNCT bracis-19024 81 38 with with ADP bracis-19024 81 39 no no DET bracis-19024 81 40 jobs job NOUN bracis-19024 81 41 in in ADP bracis-19024 81 42 the the DET bracis-19024 81 43 system system NOUN bracis-19024 81 44 , , PUNCT bracis-19024 81 45 and and CCONJ bracis-19024 81 46 \(a_0 \(a_0 NOUN bracis-19024 81 47 = = PUNCT bracis-19024 81 48 \emptyset \emptyset VERB bracis-19024 81 49 \ \ NOUN bracis-19024 81 50 ) ) PUNCT bracis-19024 81 51 , , PUNCT bracis-19024 81 52 since since SCONJ bracis-19024 81 53 there there PRON bracis-19024 81 54 is be VERB bracis-19024 81 55 no no DET bracis-19024 81 56 job job NOUN bracis-19024 81 57 to to PART bracis-19024 81 58 schedule schedule VERB bracis-19024 81 59 . . PUNCT bracis-19024 82 1 recall recall VERB bracis-19024 82 2 our our PRON bracis-19024 82 3 discussion discussion NOUN bracis-19024 82 4 about about ADP bracis-19024 82 5 classifying classify VERB bracis-19024 82 6 jobs job NOUN bracis-19024 82 7 to to PART bracis-19024 82 8 be be AUX bracis-19024 82 9 processed process VERB bracis-19024 82 10 by by ADP bracis-19024 82 11 different different ADJ bracis-19024 82 12 policies policy NOUN bracis-19024 82 13 , , PUNCT bracis-19024 82 14 versus versus ADP bracis-19024 82 15 choosing choose VERB bracis-19024 82 16 the the DET bracis-19024 82 17 next next ADJ bracis-19024 82 18 job job NOUN bracis-19024 82 19 to to PART bracis-19024 82 20 enter enter VERB bracis-19024 82 21 the the DET bracis-19024 82 22 system system NOUN bracis-19024 82 23 . . PUNCT bracis-19024 83 1 in in ADP bracis-19024 83 2 this this DET bracis-19024 83 3 paper paper NOUN bracis-19024 83 4 , , PUNCT bracis-19024 83 5 we we PRON bracis-19024 83 6 are be AUX bracis-19024 83 7 modeling model VERB bracis-19024 83 8 an an DET bracis-19024 83 9 mdp mdp NOUN bracis-19024 83 10 in in ADP bracis-19024 83 11 which which PRON bracis-19024 83 12 the the DET bracis-19024 83 13 next next ADJ bracis-19024 83 14 job job NOUN bracis-19024 83 15 is be AUX bracis-19024 83 16 chosen choose VERB bracis-19024 83 17 by by ADP bracis-19024 83 18 the the DET bracis-19024 83 19 agent agent NOUN bracis-19024 83 20 , , PUNCT bracis-19024 83 21 so so ADV bracis-19024 83 22 the the DET bracis-19024 83 23 agent agent NOUN bracis-19024 83 24 is be AUX bracis-19024 83 25 learning learn VERB bracis-19024 83 26 a a DET bracis-19024 83 27 scheduling scheduling NOUN bracis-19024 83 28 policy policy NOUN bracis-19024 83 29 . . PUNCT bracis-19024 84 1 in in ADP bracis-19024 84 2 our our PRON bracis-19024 84 3 example example NOUN bracis-19024 84 4 , , PUNCT bracis-19024 84 5 one one PRON bracis-19024 84 6 can can AUX bracis-19024 84 7 obtain obtain VERB bracis-19024 84 8 a a DET bracis-19024 84 9 reward reward NOUN bracis-19024 84 10 function function NOUN bracis-19024 84 11 by by ADP bracis-19024 84 12 using use VERB bracis-19024 84 13 the the DET bracis-19024 84 14 sequential sequential ADJ bracis-19024 84 15 version version NOUN bracis-19024 84 16 of of ADP bracis-19024 84 17 slowdown slowdown NOUN bracis-19024 84 18 , , PUNCT bracis-19024 84 19 shown show VERB bracis-19024 84 20 in in ADP bracis-19024 84 21 the the DET bracis-19024 84 22 rightmost rightmost ADJ bracis-19024 84 23 equality equality NOUN bracis-19024 84 24 of of ADP bracis-19024 84 25     SPACE bracis-19024 84 26 ( ( PUNCT bracis-19024 84 27 1 1 NUM bracis-19024 84 28 ) ) PUNCT bracis-19024 84 29 , , PUNCT bracis-19024 84 30 such such ADJ bracis-19024 84 31 that that SCONJ bracis-19024 84 32 the the DET bracis-19024 84 33 reward reward NOUN bracis-19024 84 34 at at ADP bracis-19024 84 35 each each DET bracis-19024 84 36 time time NOUN bracis-19024 84 37 step step NOUN bracis-19024 84 38 is be AUX bracis-19024 84 39 given give VERB bracis-19024 84 40 by by ADP bracis-19024 84 41 the the DET bracis-19024 84 42 sum sum NOUN bracis-19024 84 43 of of ADP bracis-19024 84 44 the the DET bracis-19024 84 45 current current ADJ bracis-19024 84 46 slowdown slowdown NOUN bracis-19024 84 47 for for ADP bracis-19024 84 48 all all DET bracis-19024 84 49 jobs job NOUN bracis-19024 84 50 in in ADP bracis-19024 84 51 the the DET bracis-19024 84 52 system system NOUN bracis-19024 84 53 : : PUNCT bracis-19024 84 54 . . PUNCT bracis-19024 85 1 when when SCONJ bracis-19024 85 2 the the DET bracis-19024 85 3 reward reward NOUN bracis-19024 85 4 function function NOUN bracis-19024 85 5 is be AUX bracis-19024 85 6 such such ADJ bracis-19024 85 7 that that SCONJ bracis-19024 85 8 it it PRON bracis-19024 85 9 computes compute VERB bracis-19024 85 10 the the DET bracis-19024 85 11 online online ADJ bracis-19024 85 12 version version NOUN bracis-19024 85 13 of of ADP bracis-19024 85 14 slowdown slowdown NOUN bracis-19024 85 15 for for ADP bracis-19024 85 16 all all DET bracis-19024 85 17 jobs job NOUN bracis-19024 85 18 in in ADP bracis-19024 85 19 the the DET bracis-19024 85 20 system system NOUN bracis-19024 85 21 , , PUNCT bracis-19024 85 22 if if SCONJ bracis-19024 85 23 \(a_1=\emptyset \(a_1=\emptyset VERB bracis-19024 85 24 \ \ NOUN bracis-19024 85 25 ) ) PUNCT bracis-19024 85 26 , , PUNCT bracis-19024 85 27 . . PUNCT bracis-19024 86 1 moreover moreover ADV bracis-19024 86 2 , , PUNCT bracis-19024 86 3 if if SCONJ bracis-19024 86 4 jobs job NOUN bracis-19024 86 5 \(j_1\ \(j_1\ VERB bracis-19024 86 6 ) ) PUNCT bracis-19024 86 7 , , PUNCT bracis-19024 86 8 \(j_2\ \(j_2\ PROPN bracis-19024 86 9 ) ) PUNCT bracis-19024 86 10 , , PUNCT bracis-19024 86 11 and and CCONJ bracis-19024 86 12 \(j_3\ \(j_3\ NOUN bracis-19024 86 13 ) ) PUNCT bracis-19024 86 14 are be AUX bracis-19024 86 15 chosen choose VERB bracis-19024 86 16 in in ADP bracis-19024 86 17 sequence sequence NOUN bracis-19024 86 18 , , PUNCT bracis-19024 86 19 the the DET bracis-19024 86 20 next next ADJ bracis-19024 86 21 state state NOUN bracis-19024 86 22 , , PUNCT bracis-19024 86 23 shown show VERB bracis-19024 86 24 in in ADP bracis-19024 86 25 fig fig NOUN bracis-19024 86 26 . . PUNCT bracis-19024 86 27     SPACE bracis-19024 87 1 1 1 NUM bracis-19024 87 2 , , PUNCT bracis-19024 87 3 will will AUX bracis-19024 87 4 be be AUX bracis-19024 87 5 given give VERB bracis-19024 87 6 by by ADP bracis-19024 87 7 sequentially sequentially ADV bracis-19024 87 8 applying apply VERB bracis-19024 87 9 the the DET bracis-19024 87 10 transition transition NOUN bracis-19024 87 11 function function NOUN bracis-19024 87 12 \(\mathcal \(\mathcal ADJ bracis-19024 87 13 { { PUNCT bracis-19024 87 14 t}\ t}\ NOUN bracis-19024 87 15 ) ) PUNCT bracis-19024 87 16 as as ADP bracis-19024 87 17 . . PUNCT bracis-19024 88 1 if if SCONJ bracis-19024 88 2 the the DET bracis-19024 88 3 episode episode NOUN bracis-19024 88 4 finished finish VERB bracis-19024 88 5 immediately immediately ADV bracis-19024 88 6 after after SCONJ bracis-19024 88 7 the the DET bracis-19024 88 8 state state NOUN bracis-19024 88 9 shown show VERB bracis-19024 88 10 in in ADP bracis-19024 88 11 fig fig NOUN bracis-19024 88 12 . . PUNCT bracis-19024 88 13     SPACE bracis-19024 88 14 1 1 NUM bracis-19024 88 15 , , PUNCT bracis-19024 88 16 the the DET bracis-19024 88 17 trajectory trajectory NOUN bracis-19024 88 18 \(\tau \(\tau PROPN bracis-19024 89 1 _ _ NOUN bracis-19024 89 2 1\ 1\ NUM bracis-19024 89 3 ) ) PUNCT bracis-19024 89 4 would would AUX bracis-19024 89 5 be be AUX bracis-19024 89 6 given give VERB bracis-19024 89 7 by by ADP bracis-19024 89 8 footnote footnote NOUN bracis-19024 89 9 3 3 NUM bracis-19024 89 10 . . PUNCT bracis-19024 90 1 the the DET bracis-19024 90 2 reward reward NOUN bracis-19024 90 3 signal signal NOUN bracis-19024 90 4 encodes encode NOUN bracis-19024 90 5 all all PRON bracis-19024 90 6 of of ADP bracis-19024 90 7 the the DET bracis-19024 90 8 agent agent NOUN bracis-19024 90 9 ’s ’s PART bracis-19024 90 10 goals goal NOUN bracis-19024 90 11 and and CCONJ bracis-19024 90 12 purposes purpose NOUN bracis-19024 90 13 , , PUNCT bracis-19024 90 14 and and CCONJ bracis-19024 90 15 the the DET bracis-19024 90 16 agent agent NOUN bracis-19024 90 17 ’s ’s PART bracis-19024 90 18 sole sole ADJ bracis-19024 90 19 objective objective NOUN bracis-19024 90 20 is be AUX bracis-19024 90 21 to to PART bracis-19024 90 22 find find VERB bracis-19024 90 23 a a DET bracis-19024 90 24 policy policy NOUN bracis-19024 90 25 \(\pi \(\pi ADP bracis-19024 90 26 _ _ PUNCT bracis-19024 90 27 \theta \theta PROPN bracis-19024 90 28 \ \ PROPN bracis-19024 90 29 ) ) PUNCT bracis-19024 90 30 parameterized parameterized ADJ bracis-19024 90 31 by by ADP bracis-19024 90 32 \(\theta \(\theta NOUN bracis-19024 90 33 \ \ NOUN bracis-19024 90 34 ) ) PUNCT bracis-19024 90 35 that that PRON bracis-19024 90 36 maximizes maximize VERB bracis-19024 90 37 the the DET bracis-19024 90 38 expected expect VERB bracis-19024 90 39 return return NOUN bracis-19024 90 40 $ $ SYM bracis-19024 90 41 $ $ SYM bracis-19024 90 42 \begin{aligned \begin{aligne VERB bracis-19024 90 43 } } PUNCT bracis-19024 90 44 g(\tau g(\tau PROPN bracis-19024 90 45 ) ) PUNCT bracis-19024 90 46 = = SYM bracis-19024 91 1 r_{1 r_{1 PROPN bracis-19024 91 2 } } PUNCT bracis-19024 91 3 + + CCONJ bracis-19024 91 4 \gamma \gamma VERB bracis-19024 91 5 r_{2 r_{2 ADV bracis-19024 91 6 } } PUNCT bracis-19024 91 7 + + NUM bracis-19024 91 8 \cdots \cdot NOUN bracis-19024 91 9 + + CCONJ bracis-19024 91 10 \gamma \gamma NUM bracis-19024 91 11 ^{t-1 ^{t-1 NOUN bracis-19024 91 12 } } PUNCT bracis-19024 91 13 r_{t r_{t PROPN bracis-19024 91 14 } } PUNCT bracis-19024 91 15 = = SYM bracis-19024 91 16 \sum \sum NOUN bracis-19024 91 17 _ _ PUNCT bracis-19024 91 18 { { PUNCT bracis-19024 91 19 t=0}^{t-1}\gamma t=0}^{t-1}\gamma PROPN bracis-19024 91 20 ^{t}r(s_t ^{t}r(s_t PROPN bracis-19024 91 21 , , PUNCT bracis-19024 91 22 a_t\sim a_t\sim NOUN bracis-19024 91 23 \pi \pi PROPN bracis-19024 91 24 _ _ PROPN bracis-19024 92 1 \theta \theta PROPN bracis-19024 92 2 ( ( PUNCT bracis-19024 92 3 s_t))\text s_t))\text PROPN bracis-19024 92 4 { { PUNCT bracis-19024 92 5 , , PUNCT bracis-19024 92 6 } } PUNCT bracis-19024 92 7 \end{aligned}$$ \end{aligned}$$ X bracis-19024 92 8 ( ( PUNCT bracis-19024 92 9 2 2 NUM bracis-19024 92 10 ) ) PUNCT bracis-19024 92 11 which which PRON bracis-19024 92 12 is be AUX bracis-19024 92 13 the the DET bracis-19024 92 14 sum sum NOUN bracis-19024 92 15 of of ADP bracis-19024 92 16 discounted discount VERB bracis-19024 92 17 rewards reward NOUN bracis-19024 92 18 encountered encounter VERB bracis-19024 92 19 by by ADP bracis-19024 92 20 the the DET bracis-19024 92 21 agent agent NOUN bracis-19024 92 22 . . PUNCT bracis-19024 93 1 when when SCONJ bracis-19024 93 2 t t PROPN bracis-19024 93 3 is be AUX bracis-19024 93 4 unbounded unbounde VERB bracis-19024 93 5 , , PUNCT bracis-19024 93 6 \(\gamma \(\gamma ADP bracis-19024 93 7 < < X bracis-19024 93 8 1\ 1\ NUM bracis-19024 93 9 ) ) PUNCT bracis-19024 93 10 . . PUNCT bracis-19024 94 1 otherwise otherwise ADV bracis-19024 94 2 , , PUNCT bracis-19024 94 3 eq eq PROPN bracis-19024 94 4 . . PROPN bracis-19024 94 5     SPACE bracis-19024 94 6 ( ( PUNCT bracis-19024 94 7 2 2 X bracis-19024 94 8 ) ) PUNCT bracis-19024 94 9 would would AUX bracis-19024 94 10 diverge diverge VERB bracis-19024 94 11 . . PUNCT bracis-19024 95 1 in in ADP bracis-19024 95 2 our our PRON bracis-19024 95 3 example example NOUN bracis-19024 95 4 , , PUNCT bracis-19024 95 5 a a DET bracis-19024 95 6 deterministic deterministic ADJ bracis-19024 95 7 policy policy NOUN bracis-19024 95 8 that that PRON bracis-19024 95 9 always always ADV bracis-19024 95 10 chose choose VERB bracis-19024 95 11 the the DET bracis-19024 95 12 smallest small ADJ bracis-19024 95 13 job job NOUN bracis-19024 95 14 first first ADV bracis-19024 95 15 would would AUX bracis-19024 95 16 yield yield VERB bracis-19024 95 17 , , PUNCT bracis-19024 95 18 while while SCONJ bracis-19024 95 19 a a DET bracis-19024 95 20 stochastic stochastic ADJ bracis-19024 95 21 policy policy NOUN bracis-19024 95 22 would would AUX bracis-19024 95 23 assign assign VERB bracis-19024 95 24 a a DET bracis-19024 95 25 probability probability NOUN bracis-19024 95 26 to to ADP bracis-19024 95 27 each each DET bracis-19024 95 28 job job NOUN bracis-19024 95 29 , , PUNCT bracis-19024 95 30 and and CCONJ bracis-19024 95 31 either either CCONJ bracis-19024 95 32 choose choose VERB bracis-19024 95 33 the the DET bracis-19024 95 34 one one NOUN bracis-19024 95 35 with with ADP bracis-19024 95 36 highest high ADJ bracis-19024 95 37 probability probability NOUN bracis-19024 95 38 or or CCONJ bracis-19024 95 39 sample sample NOUN bracis-19024 95 40 from from ADP bracis-19024 95 41 the the DET bracis-19024 95 42 jobs job NOUN bracis-19024 95 43 according accord VERB bracis-19024 95 44 to to ADP bracis-19024 95 45 that that DET bracis-19024 95 46 distribution distribution NOUN bracis-19024 95 47 . . PUNCT bracis-19024 96 1 in in ADP bracis-19024 96 2 practice practice NOUN bracis-19024 96 3 , , PUNCT bracis-19024 96 4 when when SCONJ bracis-19024 96 5 neural neural ADJ bracis-19024 96 6 networks network NOUN bracis-19024 96 7 are be AUX bracis-19024 96 8 used use VERB bracis-19024 96 9 for for ADP bracis-19024 96 10 approximation approximation NOUN bracis-19024 96 11 , , PUNCT bracis-19024 96 12 the the DET bracis-19024 96 13 last last ADJ bracis-19024 96 14 layer layer NOUN bracis-19024 96 15 of of ADP bracis-19024 96 16 the the DET bracis-19024 96 17 neural neural ADJ bracis-19024 96 18 network network NOUN bracis-19024 96 19 is be AUX bracis-19024 96 20 usually usually ADV bracis-19024 96 21 a a DET bracis-19024 96 22 softmax softmax NOUN bracis-19024 96 23 , , PUNCT bracis-19024 96 24 so so SCONJ bracis-19024 96 25 that that SCONJ bracis-19024 96 26 each each DET bracis-19024 96 27 action action NOUN bracis-19024 96 28 gets get VERB bracis-19024 96 29 a a DET bracis-19024 96 30 number number NOUN bracis-19024 96 31 that that PRON bracis-19024 96 32 can can AUX bracis-19024 96 33 be be AUX bracis-19024 96 34 interpreted interpret VERB bracis-19024 96 35 as as ADP bracis-19024 96 36 a a DET bracis-19024 96 37 probability probability NOUN bracis-19024 96 38 . . PUNCT bracis-19024 97 1 as as SCONJ bracis-19024 97 2 mentioned mention VERB bracis-19024 97 3 before before ADV bracis-19024 97 4 , , PUNCT bracis-19024 97 5 in in ADP bracis-19024 97 6 the the DET bracis-19024 97 7 example example NOUN bracis-19024 97 8 in in ADP bracis-19024 97 9 this this DET bracis-19024 97 10 section section NOUN bracis-19024 97 11 , , PUNCT bracis-19024 97 12 rewards reward NOUN bracis-19024 97 13 are be AUX bracis-19024 97 14 based base VERB bracis-19024 97 15 on on ADP bracis-19024 97 16 the the DET bracis-19024 97 17 negative negative ADJ bracis-19024 97 18 online online ADJ bracis-19024 97 19 slowdown slowdown NOUN bracis-19024 97 20 and and CCONJ bracis-19024 97 21 , , PUNCT bracis-19024 97 22 therefore therefore ADV bracis-19024 97 23 , , PUNCT bracis-19024 97 24 returns return NOUN bracis-19024 97 25 will will AUX bracis-19024 97 26 also also ADV bracis-19024 97 27 depend depend VERB bracis-19024 97 28 on on ADP bracis-19024 97 29 the the DET bracis-19024 97 30 slowdown slowdown NOUN bracis-19024 97 31 . . PUNCT bracis-19024 98 1 the the DET bracis-19024 98 2 policy policy NOUN bracis-19024 98 3 \(\pi \(\pi ADP bracis-19024 98 4 \ \ NOUN bracis-19024 98 5 ) ) PUNCT bracis-19024 98 6 is be AUX bracis-19024 98 7 a a DET bracis-19024 98 8 mapping mapping NOUN bracis-19024 98 9 from from ADP bracis-19024 98 10 states state NOUN bracis-19024 98 11 and and CCONJ bracis-19024 98 12 actions action NOUN bracis-19024 98 13 to to ADP bracis-19024 98 14 a a DET bracis-19024 98 15 probability probability NOUN bracis-19024 98 16 of of ADP bracis-19024 98 17 taking take VERB bracis-19024 98 18 an an DET bracis-19024 98 19 action action NOUN bracis-19024 98 20 a a DET bracis-19024 98 21 when when SCONJ bracis-19024 98 22 in in ADP bracis-19024 98 23 state state NOUN bracis-19024 98 24 s s NOUN bracis-19024 98 25 , , PUNCT bracis-19024 98 26 and and CCONJ bracis-19024 98 27 the the DET bracis-19024 98 28 parameters parameter NOUN bracis-19024 98 29 \(\theta \(\theta NOUN bracis-19024 98 30 \ \ PROPN bracis-19024 98 31 ) ) PUNCT bracis-19024 98 32 relate relate VERB bracis-19024 98 33 to to ADP bracis-19024 98 34 the the DET bracis-19024 98 35 approximation approximation NOUN bracis-19024 98 36 method method NOUN bracis-19024 98 37 used use VERB bracis-19024 98 38 by by ADP bracis-19024 98 39 the the DET bracis-19024 98 40 policyfootnote policyfootnote NOUN bracis-19024 98 41 4 4 NUM bracis-19024 98 42 . . X bracis-19024 98 43 popular popular ADJ bracis-19024 98 44 function function NOUN bracis-19024 98 45 approximators approximator NOUN bracis-19024 98 46 include include VERB bracis-19024 98 47 linear linear ADJ bracis-19024 98 48 combinations combination NOUN bracis-19024 98 49 of of ADP bracis-19024 98 50 features feature NOUN bracis-19024 98 51     SPACE bracis-19024 99 1 [ [ X bracis-19024 99 2 6 6 NUM bracis-19024 99 3 ] ] PUNCT bracis-19024 99 4 and and CCONJ bracis-19024 99 5 neural neural ADJ bracis-19024 99 6 networks network NOUN bracis-19024 99 7     SPACE bracis-19024 100 1 [ [ X bracis-19024 100 2 13 13 NUM bracis-19024 100 3 , , PUNCT bracis-19024 100 4 15 15 NUM bracis-19024 100 5 ] ] PUNCT bracis-19024 100 6 . . PUNCT bracis-19024 101 1 3.3 3.3 NUM bracis-19024 101 2 policy policy NOUN bracis-19024 101 3 gradients gradient NOUN bracis-19024 101 4 in in ADP bracis-19024 101 5 this this DET bracis-19024 101 6 section section NOUN bracis-19024 101 7 we we PRON bracis-19024 101 8 present present VERB bracis-19024 101 9 the the DET bracis-19024 101 10 main main ADJ bracis-19024 101 11 optimization optimization NOUN bracis-19024 101 12 method method NOUN bracis-19024 101 13 we we PRON bracis-19024 101 14 use use VERB bracis-19024 101 15 to to PART bracis-19024 101 16 find find VERB bracis-19024 101 17 policies policy NOUN bracis-19024 101 18 : : PUNCT bracis-19024 101 19 policy policy NOUN bracis-19024 101 20 gradients gradient NOUN bracis-19024 101 21 . . PUNCT bracis-19024 102 1 as as SCONJ bracis-19024 102 2 implied imply VERB bracis-19024 102 3 by by ADP bracis-19024 102 4 the the DET bracis-19024 102 5 name name NOUN bracis-19024 102 6 , , PUNCT bracis-19024 102 7 we we PRON bracis-19024 102 8 compute compute VERB bracis-19024 102 9 gradients gradient NOUN bracis-19024 102 10 of of ADP bracis-19024 102 11 policy policy NOUN bracis-19024 102 12 approximations approximation NOUN bracis-19024 102 13 , , PUNCT bracis-19024 102 14 and and CCONJ bracis-19024 102 15 use use VERB bracis-19024 102 16 them they PRON bracis-19024 102 17 to to PART bracis-19024 102 18 find find VERB bracis-19024 102 19 better well ADJ bracis-19024 102 20 parameters parameter NOUN bracis-19024 102 21 for for ADP bracis-19024 102 22 those those DET bracis-19024 102 23 functions function NOUN bracis-19024 102 24 . . PUNCT bracis-19024 103 1 formally formally ADV bracis-19024 103 2 , , PUNCT bracis-19024 103 3 we we PRON bracis-19024 103 4 generalize generalize VERB bracis-19024 103 5 policies policy NOUN bracis-19024 103 6 to to PART bracis-19024 103 7 define define VERB bracis-19024 103 8 distributions distribution NOUN bracis-19024 103 9 over over ADP bracis-19024 103 10 trajectories trajectory NOUN bracis-19024 103 11 with with ADP bracis-19024 103 12 $ $ SYM bracis-19024 103 13 $ $ SYM bracis-19024 103 14 \begin{aligned \begin{aligne VERB bracis-19024 103 15 } } PUNCT bracis-19024 103 16 \phi \phi PROPN bracis-19024 103 17 _ _ PUNCT bracis-19024 103 18 \theta \theta PROPN bracis-19024 103 19 ( ( PUNCT bracis-19024 103 20 \tau \tau PROPN bracis-19024 103 21 ) ) PUNCT bracis-19024 103 22 = = PUNCT bracis-19024 104 1 \rho \rho NOUN bracis-19024 104 2 ( ( PUNCT bracis-19024 104 3 s_0)\prod s_0)\prod X bracis-19024 104 4 _ _ PUNCT bracis-19024 104 5 t\pi t\pi NOUN bracis-19024 104 6 _ _ PUNCT bracis-19024 105 1 \theta \theta PROPN bracis-19024 105 2 ( ( PUNCT bracis-19024 105 3 a_t|s_t a_t|s_t PROPN bracis-19024 105 4 ) ) PUNCT bracis-19024 105 5 \underbrace{\mathcal \underbrace{\mathcal ADP bracis-19024 105 6 { { PUNCT bracis-19024 105 7 t}(s_{t+1}|s_t t}(s_{t+1}|s_t PROPN bracis-19024 105 8 , , PUNCT bracis-19024 105 9 a_t)}_{\text a_t)}_{\text PROPN bracis-19024 105 10 { { PUNCT bracis-19024 105 11 environment}}\text environment}}\text PROPN bracis-19024 105 12 { { PUNCT bracis-19024 105 13 , , PUNCT bracis-19024 105 14 } } PUNCT bracis-19024 105 15 \end{aligned}$$ \end{aligned}$$ X bracis-19024 105 16 ( ( PUNCT bracis-19024 105 17 3 3 NUM bracis-19024 105 18 ) ) PUNCT bracis-19024 105 19 in in ADP bracis-19024 105 20 which which PRON bracis-19024 105 21 \(\pi \(\pi PROPN bracis-19024 105 22 _ _ PUNCT bracis-19024 105 23 \theta \theta PROPN bracis-19024 105 24 \ \ PROPN bracis-19024 105 25 ) ) PUNCT bracis-19024 105 26 is be AUX bracis-19024 105 27 being be AUX bracis-19024 105 28 optimized optimize VERB bracis-19024 105 29 by by ADP bracis-19024 105 30 the the DET bracis-19024 105 31 agent agent NOUN bracis-19024 105 32 , , PUNCT bracis-19024 105 33 and and CCONJ bracis-19024 105 34 \(\rho \(\rho ADJ bracis-19024 105 35 \ \ NOUN bracis-19024 105 36 ) ) PUNCT bracis-19024 105 37 and and CCONJ bracis-19024 105 38 \(\mathcal \(\mathcal ADJ bracis-19024 105 39 { { PUNCT bracis-19024 105 40 t}\ t}\ NOUN bracis-19024 105 41 ) ) PUNCT bracis-19024 105 42 are be AUX bracis-19024 105 43 provided provide VERB bracis-19024 105 44 by by ADP bracis-19024 105 45 the the DET bracis-19024 105 46 environment environment NOUN bracis-19024 105 47 . . PUNCT bracis-19024 106 1 what what PRON bracis-19024 106 2     SPACE bracis-19024 106 3 ( ( PUNCT bracis-19024 106 4 3 3 X bracis-19024 106 5 ) ) PUNCT bracis-19024 106 6 says say VERB bracis-19024 106 7 is be AUX bracis-19024 106 8 that that SCONJ bracis-19024 106 9 we we PRON bracis-19024 106 10 can can AUX bracis-19024 106 11 assign assign VERB bracis-19024 106 12 probabilities probability NOUN bracis-19024 106 13 to to ADP bracis-19024 106 14 any any DET bracis-19024 106 15 trajectory trajectory NOUN bracis-19024 106 16 , , PUNCT bracis-19024 106 17 since since SCONJ bracis-19024 106 18 we we PRON bracis-19024 106 19 know know VERB bracis-19024 106 20 the the DET bracis-19024 106 21 distribution distribution NOUN bracis-19024 106 22 of of ADP bracis-19024 106 23 initial initial ADJ bracis-19024 106 24 states state NOUN bracis-19024 106 25 \(\rho \(\rho NOUN bracis-19024 106 26 \ \ NOUN bracis-19024 106 27 ) ) PUNCT bracis-19024 106 28 , , PUNCT bracis-19024 106 29 and and CCONJ bracis-19024 106 30 we we PRON bracis-19024 106 31 know know VERB bracis-19024 106 32 that that SCONJ bracis-19024 106 33 the the DET bracis-19024 106 34 policy policy NOUN bracis-19024 106 35 will will AUX bracis-19024 106 36 assign assign VERB bracis-19024 106 37 probabilities probability NOUN bracis-19024 106 38 to to ADP bracis-19024 106 39 actions action NOUN bracis-19024 106 40 given give VERB bracis-19024 106 41 states state NOUN bracis-19024 106 42 , , PUNCT bracis-19024 106 43 and and CCONJ bracis-19024 106 44 that that SCONJ bracis-19024 106 45 , , PUNCT bracis-19024 106 46 when when SCONJ bracis-19024 106 47 such such ADJ bracis-19024 106 48 actions action NOUN bracis-19024 106 49 are be AUX bracis-19024 106 50 taken take VERB bracis-19024 106 51 , , PUNCT bracis-19024 106 52 the the DET bracis-19024 106 53 environment environment NOUN bracis-19024 106 54 will will AUX bracis-19024 106 55 sample sample VERB bracis-19024 106 56 a a DET bracis-19024 106 57 new new ADJ bracis-19024 106 58 state state NOUN bracis-19024 106 59 for for ADP bracis-19024 106 60 the the DET bracis-19024 106 61 agent agent NOUN bracis-19024 106 62 . . PUNCT bracis-19024 107 1 when when SCONJ bracis-19024 107 2 we we PRON bracis-19024 107 3 do do VERB bracis-19024 107 4 so so ADV bracis-19024 107 5 , , PUNCT bracis-19024 107 6 we we PRON bracis-19024 107 7 can can AUX bracis-19024 107 8 define define VERB bracis-19024 107 9 an an DET bracis-19024 107 10 optimization optimization NOUN bracis-19024 107 11 objective objective NOUN bracis-19024 107 12 to to PART bracis-19024 107 13 find find VERB bracis-19024 107 14 the the DET bracis-19024 107 15 optimal optimal ADJ bracis-19024 107 16 set set NOUN bracis-19024 107 17 of of ADP bracis-19024 107 18 parameters parameter NOUN bracis-19024 107 19 $ $ SYM bracis-19024 107 20 $ $ SYM bracis-19024 107 21 \begin{aligned \begin{aligne VERB bracis-19024 107 22 } } PUNCT bracis-19024 107 23 \theta \theta NOUN bracis-19024 107 24 ^ ^ SYM bracis-19024 107 25 * * PUNCT bracis-19024 107 26 = = PUNCT bracis-19024 107 27 \arg \arg ADP bracis-19024 107 28 \max \max PROPN bracis-19024 107 29 _ _ PUNCT bracis-19024 108 1 \theta \theta PROPN bracis-19024 108 2 j(\theta j(\theta PROPN bracis-19024 108 3 ) ) PUNCT bracis-19024 109 1 = = PUNCT bracis-19024 109 2 \arg \arg ADP bracis-19024 109 3 \max \max PROPN bracis-19024 109 4 _ _ PUNCT bracis-19024 110 1 \theta \theta PROPN bracis-19024 110 2 \int \int VERB bracis-19024 110 3 _ _ PUNCT bracis-19024 110 4 \tau \tau PROPN bracis-19024 110 5 g(\tau g(\tau PROPN bracis-19024 110 6 ) ) PUNCT bracis-19024 111 1 \phi \phi PROPN bracis-19024 111 2 _ _ PUNCT bracis-19024 112 1 \theta \theta PROPN bracis-19024 112 2 ( ( PUNCT bracis-19024 112 3 \tau \tau PROPN bracis-19024 112 4 ) ) PUNCT bracis-19024 113 1 d\tau d\tau PROPN bracis-19024 114 1 \text \text NOUN bracis-19024 114 2 { { PUNCT bracis-19024 114 3 , , PUNCT bracis-19024 114 4 } } PUNCT bracis-19024 114 5 \end{aligned}$$ \end{aligned}$$ X bracis-19024 114 6 ( ( PUNCT bracis-19024 114 7 4 4 NUM bracis-19024 114 8 ) ) PUNCT bracis-19024 114 9 where where SCONJ bracis-19024 114 10 \(j(\theta \(j(\theta X bracis-19024 114 11 ) ) PUNCT bracis-19024 114 12 \ \ NOUN bracis-19024 114 13 ) ) PUNCT bracis-19024 114 14 is be AUX bracis-19024 114 15 the the DET bracis-19024 114 16 performance performance NOUN bracis-19024 114 17 measure measure NOUN bracis-19024 114 18 given give VERB bracis-19024 114 19 by by ADP bracis-19024 114 20 the the DET bracis-19024 114 21 expected expect VERB bracis-19024 114 22 return return NOUN bracis-19024 114 23 of of ADP bracis-19024 114 24 a a DET bracis-19024 114 25 trajectory trajectory NOUN bracis-19024 114 26 , , PUNCT bracis-19024 114 27 which which PRON bracis-19024 114 28 can can AUX bracis-19024 114 29 be be AUX bracis-19024 114 30 approximated approximate VERB bracis-19024 114 31 by by ADP bracis-19024 114 32 a a DET bracis-19024 114 33 monte monte PROPN bracis-19024 114 34 carlo carlo PROPN bracis-19024 114 35 estimate estimate NOUN bracis-19024 114 36 footnote footnote VERB bracis-19024 114 37 5 5 NUM bracis-19024 114 38 . . PUNCT bracis-19024 115 1 if if SCONJ bracis-19024 115 2 we we PRON bracis-19024 115 3 construct construct VERB bracis-19024 115 4 \(\widehat{j(\theta \(\widehat{j(\theta PROPN bracis-19024 115 5 ) ) PUNCT bracis-19024 115 6 } } PUNCT bracis-19024 115 7 \ \ NOUN bracis-19024 115 8 ) ) PUNCT bracis-19024 115 9 such such ADJ bracis-19024 115 10 that that SCONJ bracis-19024 115 11 it it PRON bracis-19024 115 12 is be AUX bracis-19024 115 13 differentiable differentiable ADJ bracis-19024 115 14 , , PUNCT bracis-19024 115 15 we we PRON bracis-19024 115 16 can can AUX bracis-19024 115 17 approximate approximate VERB bracis-19024 115 18 \(\theta \(\theta PROPN bracis-19024 115 19 ^*\ ^*\ NUM bracis-19024 115 20 ) ) PUNCT bracis-19024 115 21 by by ADP bracis-19024 115 22 gradient gradient NOUN bracis-19024 115 23 ascent ascent NOUN bracis-19024 115 24 in in ADP bracis-19024 115 25 \(\theta \(\theta NOUN bracis-19024 115 26 \ \ PROPN bracis-19024 115 27 ) ) PUNCT bracis-19024 115 28 , , PUNCT bracis-19024 115 29 such such ADJ bracis-19024 115 30 that that DET bracis-19024 115 31 \(\theta \(\theta NOUN bracis-19024 115 32 _ _ PRON bracis-19024 115 33 { { PUNCT bracis-19024 115 34 j+1}\leftarrow j+1}\leftarrow NOUN bracis-19024 115 35 \theta \theta PROPN bracis-19024 115 36 _ _ PROPN bracis-19024 115 37 j+\alpha j+\alpha PUNCT bracis-19024 115 38 \nabla \nabla PROPN bracis-19024 115 39 \widehat{j(\theta \widehat{j(\theta PROPN bracis-19024 115 40 ) ) PUNCT bracis-19024 115 41 } } PUNCT bracis-19024 115 42 \ \ NOUN bracis-19024 115 43 ) ) PUNCT bracis-19024 115 44 , , PUNCT bracis-19024 115 45 with with ADP bracis-19024 115 46 \(\alpha \(\alpha NOUN bracis-19024 115 47 > > X bracis-19024 115 48 0\ 0\ PROPN bracis-19024 115 49 ) ) PUNCT bracis-19024 115 50 , , PUNCT bracis-19024 115 51 yielding yield VERB bracis-19024 115 52 $ $ SYM bracis-19024 115 53 $ $ SYM bracis-19024 115 54 \begin{aligned \begin{aligne VERB bracis-19024 115 55 } } PUNCT bracis-19024 115 56 \nabla \nabla NOUN bracis-19024 115 57 _ _ PROPN bracis-19024 116 1 \theta \theta PROPN bracis-19024 116 2 j(\theta j(\theta PROPN bracis-19024 116 3 ) ) PUNCT bracis-19024 116 4 \approx \approx PROPN bracis-19024 116 5 \frac{1}{n}\sum \frac{1}{n}\sum NOUN bracis-19024 116 6 _ _ NOUN bracis-19024 117 1 i i PRON bracis-19024 117 2 g(\tau g(\tau VERB bracis-19024 117 3 _ _ NOUN bracis-19024 117 4 i)\nabla i)\nabla NOUN bracis-19024 118 1 _ _ PUNCT bracis-19024 119 1 \theta \theta PROPN bracis-19024 119 2 \log \log PROPN bracis-19024 119 3 \phi \phi PROPN bracis-19024 119 4 _ _ PUNCT bracis-19024 119 5 \theta \theta PROPN bracis-19024 119 6 ( ( PUNCT bracis-19024 119 7 \tau \tau PROPN bracis-19024 119 8 _ _ PUNCT bracis-19024 119 9 i);\tau i);\tau X bracis-19024 119 10 _ _ PUNCT bracis-19024 119 11 i\sim i\sim ADV bracis-19024 119 12 \pi \pi PROPN bracis-19024 119 13 _ _ NOUN bracis-19024 120 1 \theta \theta PROPN bracis-19024 120 2 \text \text PROPN bracis-19024 120 3 { { PUNCT bracis-19024 120 4 . . PUNCT bracis-19024 120 5 } } PUNCT bracis-19024 120 6 \end{aligned}$$ \end{aligned}$$ X bracis-19024 120 7 ( ( PUNCT bracis-19024 120 8 5 5 NUM bracis-19024 120 9 ) ) PUNCT bracis-19024 120 10 by by ADP bracis-19024 120 11 expanding expand VERB bracis-19024 120 12     SPACE bracis-19024 120 13 ( ( PUNCT bracis-19024 120 14 2 2 NUM bracis-19024 120 15 ) ) PUNCT bracis-19024 120 16 by by ADP bracis-19024 120 17 one one NUM bracis-19024 120 18 time time NOUN bracis-19024 120 19 - - PUNCT bracis-19024 120 20 step step NOUN bracis-19024 120 21 , , PUNCT bracis-19024 120 22 we we PRON bracis-19024 120 23 get get VERB bracis-19024 120 24 the the DET bracis-19024 120 25 update update NOUN bracis-19024 120 26 \(g(\tau \(g(\tau ADV bracis-19024 120 27 ) ) PUNCT bracis-19024 121 1 = = SYM bracis-19024 121 2 r_{1 r_{1 PROPN bracis-19024 121 3 } } PUNCT bracis-19024 121 4 + + CCONJ bracis-19024 121 5 \gamma \gamma ADP bracis-19024 121 6 g(\tau g(\tau PROPN bracis-19024 121 7 _ _ NOUN bracis-19024 121 8 1)\ 1)\ NUM bracis-19024 121 9 ) ) PUNCT bracis-19024 121 10 or or CCONJ bracis-19024 121 11 , , PUNCT bracis-19024 121 12 more more ADV bracis-19024 121 13 generally generally ADV bracis-19024 121 14 , , PUNCT bracis-19024 121 15 \(g(\tau \(g(\tau ADV bracis-19024 121 16 _ _ PUNCT bracis-19024 121 17 t)=r_{t+1 t)=r_{t+1 X bracis-19024 121 18 } } PUNCT bracis-19024 121 19 + + CCONJ bracis-19024 121 20 \gamma \gamma ADP bracis-19024 121 21 g(\tau g(\tau PROPN bracis-19024 121 22 _ _ NOUN bracis-19024 121 23 { { PUNCT bracis-19024 121 24 t+1})\ t+1})\ NOUN bracis-19024 121 25 ) ) PUNCT bracis-19024 121 26 , , PUNCT bracis-19024 121 27 where where SCONJ bracis-19024 121 28 \(\tau \(\tau VERB bracis-19024 121 29 _ _ PRON bracis-19024 121 30 i\ i\ NOUN bracis-19024 121 31 ) ) PUNCT bracis-19024 121 32 indicates indicate VERB bracis-19024 121 33 the the DET bracis-19024 121 34 trajectory trajectory NOUN bracis-19024 121 35 \(\tau \(\tau PUNCT bracis-19024 121 36 \ \ NOUN bracis-19024 121 37 ) ) PUNCT bracis-19024 121 38 starting start VERB bracis-19024 121 39 from from ADP bracis-19024 121 40 offset offset ADJ bracis-19024 121 41 i. i. NOUN bracis-19024 121 42 this this DET bracis-19024 121 43 update update NOUN bracis-19024 121 44 is be AUX bracis-19024 121 45 usually usually ADV bracis-19024 121 46 written write VERB bracis-19024 121 47 as as SCONJ bracis-19024 121 48 \(g_t \(g_t NOUN bracis-19024 121 49 = = SYM bracis-19024 121 50 r_{t+1}+\gamma r_{t+1}+\gamma NOUN bracis-19024 121 51 v_\pi v_\pi NOUN bracis-19024 121 52 ( ( PUNCT bracis-19024 121 53 s_{t+1})\ s_{t+1})\ NOUN bracis-19024 121 54 ) ) PUNCT bracis-19024 121 55 , , PUNCT bracis-19024 121 56 where where SCONJ bracis-19024 121 57 \(g_t\ \(g_t\ NOUN bracis-19024 121 58 ) ) PUNCT bracis-19024 121 59 is be AUX bracis-19024 121 60 shorthand shorthand NOUN bracis-19024 121 61 notation notation NOUN bracis-19024 121 62 for for ADP bracis-19024 121 63 \(g(\tau \(g(\tau ADV bracis-19024 121 64 _ _ PUNCT bracis-19024 121 65 t)\ t)\ PRON bracis-19024 121 66 ) ) PUNCT bracis-19024 121 67 , , PUNCT bracis-19024 121 68 and and CCONJ bracis-19024 121 69 \(v_\pi \(v_\pi NOUN bracis-19024 121 70 ( ( PUNCT bracis-19024 121 71 s_{t+1})\ s_{t+1})\ NOUN bracis-19024 121 72 ) ) PUNCT bracis-19024 121 73 is be AUX bracis-19024 121 74 the the DET bracis-19024 121 75 return return NOUN bracis-19024 121 76 when when SCONJ bracis-19024 121 77 starting start VERB bracis-19024 121 78 at at ADP bracis-19024 121 79 state state NOUN bracis-19024 121 80 \(s_{t+1}\ \(s_{t+1}\ NUM bracis-19024 121 81 ) ) PUNCT bracis-19024 121 82 and and CCONJ bracis-19024 121 83 following follow VERB bracis-19024 121 84 policy policy NOUN bracis-19024 121 85 \(\pi \(\pi ADP bracis-19024 121 86 \ \ NOUN bracis-19024 121 87 ) ) PUNCT bracis-19024 121 88 ( ( PUNCT bracis-19024 121 89 which which PRON bracis-19024 121 90 generated generate VERB bracis-19024 121 91 trajectory trajectory NOUN bracis-19024 121 92 \(\tau \(\tau PUNCT bracis-19024 121 93 \ \ NOUN bracis-19024 121 94 ) ) PUNCT bracis-19024 121 95 ) ) PUNCT bracis-19024 121 96 . . PUNCT bracis-19024 122 1 another another DET bracis-19024 122 2 function function NOUN bracis-19024 122 3 related relate VERB bracis-19024 122 4 to to ADP bracis-19024 122 5 \(v_\pi \(v_\pi NOUN bracis-19024 122 6 ( ( PUNCT bracis-19024 122 7 s_t)\ s_t)\ PUNCT bracis-19024 122 8 ) ) PUNCT bracis-19024 122 9 is be AUX bracis-19024 122 10 \(q_\pi \(q_\pi NOUN bracis-19024 122 11 ( ( PUNCT bracis-19024 122 12 s_t s_t NOUN bracis-19024 122 13 , , PUNCT bracis-19024 122 14 a_t)\ a_t)\ NOUN bracis-19024 122 15 ) ) PUNCT bracis-19024 122 16 , , PUNCT bracis-19024 122 17 which which PRON bracis-19024 122 18 gives give VERB bracis-19024 122 19 the the DET bracis-19024 122 20 return return NOUN bracis-19024 122 21 when when SCONJ bracis-19024 122 22 starting start VERB bracis-19024 122 23 at at ADP bracis-19024 122 24 state state NOUN bracis-19024 122 25 \(s_t\ \(s_t\ PROPN bracis-19024 122 26 ) ) PUNCT bracis-19024 122 27 and and CCONJ bracis-19024 122 28 taking take VERB bracis-19024 122 29 action action NOUN bracis-19024 122 30 \(a_t\ \(a_t\ NUM bracis-19024 122 31 ) ) PUNCT bracis-19024 122 32 then then ADV bracis-19024 122 33 following follow VERB bracis-19024 122 34 policy policy NOUN bracis-19024 122 35 \(\pi \(\pi ADP bracis-19024 122 36 \ \ NOUN bracis-19024 122 37 ) ) PUNCT bracis-19024 122 38 . . PUNCT bracis-19024 123 1 with with ADP bracis-19024 123 2 these these DET bracis-19024 123 3 two two NUM bracis-19024 123 4 functions function NOUN bracis-19024 123 5 , , PUNCT bracis-19024 123 6 we we PRON bracis-19024 123 7 can can AUX bracis-19024 123 8 define define VERB bracis-19024 123 9 a a DET bracis-19024 123 10 third third ADJ bracis-19024 123 11 one one NUM bracis-19024 123 12 , , PUNCT bracis-19024 123 13 which which PRON bracis-19024 123 14 gives give VERB bracis-19024 123 15 the the DET bracis-19024 123 16 relative relative ADJ bracis-19024 123 17 advantage advantage NOUN bracis-19024 123 18 of of ADP bracis-19024 123 19 taking take VERB bracis-19024 123 20 action action NOUN bracis-19024 123 21 \(a_t\ \(a_t\ NOUN bracis-19024 123 22 ) ) PUNCT bracis-19024 123 23 when when SCONJ bracis-19024 123 24 in in ADP bracis-19024 123 25 state state NOUN bracis-19024 123 26 \(s_t\ \(s_t\ PROPN bracis-19024 123 27 ) ) PUNCT bracis-19024 123 28 , , PUNCT bracis-19024 123 29 defined define VERB bracis-19024 123 30 as as ADP bracis-19024 123 31 \(a_\pi \(a_\pi NOUN bracis-19024 123 32 ( ( PUNCT bracis-19024 123 33 s_t s_t NOUN bracis-19024 123 34 , , PUNCT bracis-19024 123 35 a_t a_t PUNCT bracis-19024 123 36 ) ) PUNCT bracis-19024 123 37 = = SYM bracis-19024 123 38 q_\pi q_\pi X bracis-19024 123 39 ( ( PUNCT bracis-19024 123 40 s_t s_t NOUN bracis-19024 123 41 , , PUNCT bracis-19024 123 42 a_t)-v_\pi a_t)-v_\pi PROPN bracis-19024 123 43 ( ( PUNCT bracis-19024 123 44 s_t)\ s_t)\ NOUN bracis-19024 123 45 ) ) PUNCT bracis-19024 123 46 , , PUNCT bracis-19024 123 47 and and CCONJ bracis-19024 123 48 called call VERB bracis-19024 123 49 the the DET bracis-19024 123 50 advantage advantage NOUN bracis-19024 123 51 function function NOUN bracis-19024 123 52 . . PUNCT bracis-19024 124 1 as as ADP bracis-19024 124 2 with with ADP bracis-19024 124 3 the the DET bracis-19024 124 4 policy policy NOUN bracis-19024 124 5 , , PUNCT bracis-19024 124 6 \(q_\pi \(q_\pi PROPN bracis-19024 124 7 \ \ PROPN bracis-19024 124 8 ) ) PUNCT bracis-19024 124 9 , , PUNCT bracis-19024 124 10 \(v_\pi \(v_\pi X bracis-19024 124 11 \ \ NOUN bracis-19024 124 12 ) ) PUNCT bracis-19024 124 13 , , PUNCT bracis-19024 124 14 and and CCONJ bracis-19024 124 15 \(a_\pi \(a_\pi VERB bracis-19024 124 16 \ \ NOUN bracis-19024 124 17 ) ) PUNCT bracis-19024 124 18 can can AUX bracis-19024 124 19 also also ADV bracis-19024 124 20 be be AUX bracis-19024 124 21 approximated approximate VERB bracis-19024 124 22 and and CCONJ bracis-19024 124 23 , , PUNCT bracis-19024 124 24 thus thus ADV bracis-19024 124 25 , , PUNCT bracis-19024 124 26 learned learn VERB bracis-19024 124 27 . . PUNCT bracis-19024 125 1 when when SCONJ bracis-19024 125 2 such such DET bracis-19024 125 3 an an DET bracis-19024 125 4 approximation approximation NOUN bracis-19024 125 5 is be AUX bracis-19024 125 6 used use VERB bracis-19024 125 7 , , PUNCT bracis-19024 125 8 the the DET bracis-19024 125 9 update update NOUN bracis-19024 125 10     SPACE bracis-19024 125 11 ( ( PUNCT bracis-19024 125 12 5 5 NUM bracis-19024 125 13 ) ) PUNCT bracis-19024 125 14 becomes become VERB bracis-19024 125 15 $ $ SYM bracis-19024 125 16 $ $ SYM bracis-19024 125 17 \begin{aligned \begin{aligne VERB bracis-19024 125 18 } } PUNCT bracis-19024 125 19 \nabla \nabla NOUN bracis-19024 125 20 _ _ PROPN bracis-19024 126 1 \theta \theta PROPN bracis-19024 126 2 j(\theta j(\theta PROPN bracis-19024 126 3 ) ) PUNCT bracis-19024 126 4 \approx \approx PROPN bracis-19024 126 5 \frac{1}{n}\sum \frac{1}{n}\sum NOUN bracis-19024 126 6 _ _ PUNCT bracis-19024 127 1 i i PRON bracis-19024 127 2 \nabla \nabla VERB bracis-19024 127 3 _ _ PUNCT bracis-19024 128 1 \theta \theta PROPN bracis-19024 128 2 \log \log PROPN bracis-19024 128 3 \pi \pi PROPN bracis-19024 128 4 _ _ PROPN bracis-19024 128 5 \theta \theta PROPN bracis-19024 128 6 ( ( PUNCT bracis-19024 128 7 a_t|s_t)\hat{a}_\pi a_t|s_t)\hat{a}_\pi PROPN bracis-19024 128 8 ( ( PUNCT bracis-19024 128 9 s_t s_t NOUN bracis-19024 128 10 , , PUNCT bracis-19024 128 11 a_t a_t NUM bracis-19024 128 12 ) ) PUNCT bracis-19024 128 13 ; ; PUNCT bracis-19024 128 14 s_t s_t NUM bracis-19024 128 15 , , PUNCT bracis-19024 128 16 a_t\sim a_t\sim NOUN bracis-19024 128 17 \pi \pi PROPN bracis-19024 128 18 _ _ PROPN bracis-19024 129 1 \theta \theta PROPN bracis-19024 129 2 \text \text PROPN bracis-19024 129 3 { { PUNCT bracis-19024 129 4 , , PUNCT bracis-19024 129 5 } } PUNCT bracis-19024 129 6 \end{aligned}$$ \end{aligned}$$ X bracis-19024 129 7 ( ( PUNCT bracis-19024 129 8 6 6 NUM bracis-19024 129 9 ) ) PUNCT bracis-19024 129 10 where where SCONJ bracis-19024 129 11 \(\hat{a}_\pi \(\hat{a}_\pi NOUN bracis-19024 129 12 \ \ PROPN bracis-19024 129 13 ) ) PUNCT bracis-19024 129 14 is be AUX bracis-19024 129 15 an an DET bracis-19024 129 16 approximation approximation NOUN bracis-19024 129 17 of of ADP bracis-19024 129 18 \(a_\pi \(a_\pi NOUN bracis-19024 129 19 \ \ NOUN bracis-19024 129 20 ) ) PUNCT bracis-19024 129 21 , , PUNCT bracis-19024 129 22 and and CCONJ bracis-19024 129 23 which which PRON bracis-19024 129 24 can can AUX bracis-19024 129 25 be be AUX bracis-19024 129 26 further far ADV bracis-19024 129 27 split split VERB bracis-19024 129 28 into into ADP bracis-19024 129 29 two two NUM bracis-19024 129 30 estimators estimator NOUN bracis-19024 129 31 as as ADP bracis-19024 129 32 \(\hat{a}_\pi \(\hat{a}_\pi NOUN bracis-19024 129 33 = = NOUN bracis-19024 129 34 \hat{q}_\pi \hat{q}_\pi PROPN bracis-19024 129 35 -\hat{v}_\pi -\hat{v}_\pi NOUN bracis-19024 129 36 \ \ PROPN bracis-19024 129 37 ) ) PUNCT bracis-19024 129 38 , , PUNCT bracis-19024 129 39 with with ADP bracis-19024 129 40 the the DET bracis-19024 129 41 arguments argument NOUN bracis-19024 129 42 \(s_t\ \(s_t\ ADJ bracis-19024 129 43 ) ) PUNCT bracis-19024 129 44 and and CCONJ bracis-19024 129 45 \(a_t\ \(a_t\ NUM bracis-19024 129 46 ) ) PUNCT bracis-19024 129 47 dropped drop VERB bracis-19024 129 48 for for ADP bracis-19024 129 49 better well ADJ bracis-19024 129 50 readability readability NOUN bracis-19024 129 51 . . PUNCT bracis-19024 130 1 in in ADP bracis-19024 130 2 this this DET bracis-19024 130 3 setting setting NOUN bracis-19024 130 4 , , PUNCT bracis-19024 130 5 the the DET bracis-19024 130 6 \(\pi \(\pi PROPN bracis-19024 130 7 _ _ PUNCT bracis-19024 130 8 \theta \theta PROPN bracis-19024 130 9 \ \ ADJ bracis-19024 130 10 ) ) PUNCT bracis-19024 130 11 approximator approximator NOUN bracis-19024 130 12 is be AUX bracis-19024 130 13 called call VERB bracis-19024 130 14 an an DET bracis-19024 130 15 actor actor NOUN bracis-19024 130 16 , , PUNCT bracis-19024 130 17 and and CCONJ bracis-19024 130 18 the the DET bracis-19024 130 19 \(\hat{a}_\pi \(\hat{a}_\pi ADJ bracis-19024 130 20 \ \ NOUN bracis-19024 130 21 ) ) PUNCT bracis-19024 130 22 approximator approximator NOUN bracis-19024 130 23 is be AUX bracis-19024 130 24 called call VERB bracis-19024 130 25 a a DET bracis-19024 130 26 critic critic NOUN bracis-19024 130 27 . . PUNCT bracis-19024 131 1 in in ADP bracis-19024 131 2 the the DET bracis-19024 131 3 literature literature NOUN bracis-19024 131 4 , , PUNCT bracis-19024 131 5 we we PRON bracis-19024 131 6 find find VERB bracis-19024 131 7 techniques technique NOUN bracis-19024 131 8 that that PRON bracis-19024 131 9 regularize regularize VERB bracis-19024 131 10 updates update NOUN bracis-19024 131 11     SPACE bracis-19024 131 12 [ [ X bracis-19024 131 13 8 8 NUM bracis-19024 131 14 , , PUNCT bracis-19024 131 15 12 12 NUM bracis-19024 131 16 ] ] PUNCT bracis-19024 131 17 , , PUNCT bracis-19024 131 18 but but CCONJ bracis-19024 131 19 as as SCONJ bracis-19024 131 20 presented present VERB bracis-19024 131 21 , , PUNCT bracis-19024 131 22 eq eq PROPN bracis-19024 131 23 . . PROPN bracis-19024 131 24     SPACE bracis-19024 131 25 ( ( PUNCT bracis-19024 131 26 6 6 NUM bracis-19024 131 27 ) ) PUNCT bracis-19024 131 28 is be AUX bracis-19024 131 29 sufficient sufficient ADJ bracis-19024 131 30 for for ADP bracis-19024 131 31 understanding understanding NOUN bracis-19024 131 32 of of ADP bracis-19024 131 33 the the DET bracis-19024 131 34 techniques technique NOUN bracis-19024 131 35 discussed discuss VERB bracis-19024 131 36 in in ADP bracis-19024 131 37 this this DET bracis-19024 131 38 paper paper NOUN bracis-19024 131 39 . . PUNCT bracis-19024 132 1 4 4 NUM bracis-19024 132 2 methodology methodology NOUN bracis-19024 132 3 although although SCONJ bracis-19024 132 4 the the DET bracis-19024 132 5 discussion discussion NOUN bracis-19024 132 6 in in ADP bracis-19024 132 7 the the DET bracis-19024 132 8 previous previous ADJ bracis-19024 132 9 section section NOUN bracis-19024 132 10 is be AUX bracis-19024 132 11 helpful helpful ADJ bracis-19024 132 12 for for ADP bracis-19024 132 13 conceptualizing conceptualize VERB bracis-19024 132 14 the the DET bracis-19024 132 15 problem problem NOUN bracis-19024 132 16 we we PRON bracis-19024 132 17 are be AUX bracis-19024 132 18 interested interested ADJ bracis-19024 132 19 in in ADP bracis-19024 132 20 , , PUNCT bracis-19024 132 21 it it PRON bracis-19024 132 22 is be AUX bracis-19024 132 23 not not PART bracis-19024 132 24 enough enough ADJ bracis-19024 132 25 to to PART bracis-19024 132 26 help help VERB bracis-19024 132 27 us we PRON bracis-19024 132 28 implement implement VERB bracis-19024 132 29 a a DET bracis-19024 132 30 solution solution NOUN bracis-19024 132 31 , , PUNCT bracis-19024 132 32 since since SCONJ bracis-19024 132 33 it it PRON bracis-19024 132 34 does do AUX bracis-19024 132 35 not not PART bracis-19024 132 36 specify specify VERB bracis-19024 132 37 when when SCONJ bracis-19024 132 38 the the DET bracis-19024 132 39 agent agent NOUN bracis-19024 132 40 is be AUX bracis-19024 132 41 invoked invoke VERB bracis-19024 132 42 for for ADP bracis-19024 132 43 learning learn VERB bracis-19024 132 44 , , PUNCT bracis-19024 132 45 how how SCONJ bracis-19024 132 46 a a DET bracis-19024 132 47 state state NOUN bracis-19024 132 48 is be AUX bracis-19024 132 49 actually actually ADV bracis-19024 132 50 represented represent VERB bracis-19024 132 51 , , PUNCT bracis-19024 132 52 nor nor CCONJ bracis-19024 132 53 how how SCONJ bracis-19024 132 54 rewards reward NOUN bracis-19024 132 55 are be AUX bracis-19024 132 56 computed compute VERB bracis-19024 132 57 for for ADP bracis-19024 132 58 each each DET bracis-19024 132 59 action action NOUN bracis-19024 132 60 . . PUNCT bracis-19024 133 1 in in ADP bracis-19024 133 2 this this DET bracis-19024 133 3 section section NOUN bracis-19024 133 4 , , PUNCT bracis-19024 133 5 we we PRON bracis-19024 133 6 will will AUX bracis-19024 133 7 detail detail VERB bracis-19024 133 8 our our PRON bracis-19024 133 9 design design NOUN bracis-19024 133 10 decisions decision NOUN bracis-19024 133 11 , , PUNCT bracis-19024 133 12 and and CCONJ bracis-19024 133 13 will will AUX bracis-19024 133 14 elaborate elaborate VERB bracis-19024 133 15 on on ADP bracis-19024 133 16 what what DET bracis-19024 133 17 changes change NOUN bracis-19024 133 18 are be AUX bracis-19024 133 19 required require VERB bracis-19024 133 20 to to PART bracis-19024 133 21 assess assess VERB bracis-19024 133 22 the the DET bracis-19024 133 23 impact impact NOUN bracis-19024 133 24 of of ADP bracis-19024 133 25 said say VERB bracis-19024 133 26 decisions decision NOUN bracis-19024 133 27 in in ADP bracis-19024 133 28 rl rl NOUN bracis-19024 133 29 performance performance NOUN bracis-19024 133 30 . . PUNCT bracis-19024 134 1 we we PRON bracis-19024 134 2 begin begin VERB bracis-19024 134 3 by by ADP bracis-19024 134 4 describing describe VERB bracis-19024 134 5 the the DET bracis-19024 134 6 base base PROPN bracis-19024 134 7 mdp mdp PROPN bracis-19024 134 8 , , PUNCT bracis-19024 134 9 and and CCONJ bracis-19024 134 10 then then ADV bracis-19024 134 11 we we PRON bracis-19024 134 12 will will AUX bracis-19024 134 13 describe describe VERB bracis-19024 134 14 incremental incremental ADJ bracis-19024 134 15 changes change NOUN bracis-19024 134 16 that that PRON bracis-19024 134 17 can can AUX bracis-19024 134 18 be be AUX bracis-19024 134 19 made make VERB bracis-19024 134 20 to to ADP bracis-19024 134 21 the the DET bracis-19024 134 22 environment environment NOUN bracis-19024 134 23 so so SCONJ bracis-19024 134 24 that that SCONJ bracis-19024 134 25 it it PRON bracis-19024 134 26 may may AUX bracis-19024 134 27 become become VERB bracis-19024 134 28 faster fast ADV bracis-19024 134 29 to to PART bracis-19024 134 30 compute compute VERB bracis-19024 134 31 , , PUNCT bracis-19024 134 32 and and CCONJ bracis-19024 134 33 easier easy ADJ bracis-19024 134 34 to to PART bracis-19024 134 35 learn learn VERB bracis-19024 134 36 , , PUNCT bracis-19024 134 37 leading lead VERB bracis-19024 134 38 to to ADP bracis-19024 134 39 faster fast ADJ bracis-19024 134 40 convergence convergence NOUN bracis-19024 134 41 . . PUNCT bracis-19024 135 1 we we PRON bracis-19024 135 2 implemented implement VERB bracis-19024 135 3 the the DET bracis-19024 135 4 base base NOUN bracis-19024 135 5 mdp mdp NOUN bracis-19024 135 6 and and CCONJ bracis-19024 135 7 each each DET bracis-19024 135 8 incremental incremental ADJ bracis-19024 135 9 change change NOUN bracis-19024 135 10 discussed discuss VERB bracis-19024 135 11 in in ADP bracis-19024 135 12 this this DET bracis-19024 135 13 section section NOUN bracis-19024 135 14 . . PUNCT bracis-19024 136 1 then then ADV bracis-19024 136 2 , , PUNCT bracis-19024 136 3 we we PRON bracis-19024 136 4 evaluated evaluate VERB bracis-19024 136 5 all all DET bracis-19024 136 6 implementations implementation NOUN bracis-19024 136 7 , , PUNCT bracis-19024 136 8 observing observe VERB bracis-19024 136 9 both both DET bracis-19024 136 10 convergence convergence NOUN bracis-19024 136 11 performance performance NOUN bracis-19024 136 12 and and CCONJ bracis-19024 136 13 final final ADJ bracis-19024 136 14 agent agent NOUN bracis-19024 136 15 performance performance NOUN bracis-19024 136 16 in in ADP bracis-19024 136 17 the the DET bracis-19024 136 18 task task NOUN bracis-19024 136 19 of of ADP bracis-19024 136 20 scheduling scheduling NOUN bracis-19024 136 21 hpc hpc PROPN bracis-19024 136 22 jobs job NOUN bracis-19024 136 23 . . PUNCT bracis-19024 137 1 4.1 4.1 NUM bracis-19024 137 2 the the DET bracis-19024 137 3 base base NOUN bracis-19024 137 4 , , PUNCT bracis-19024 137 5 image image NOUN bracis-19024 137 6 - - PUNCT bracis-19024 137 7 like like ADJ bracis-19024 137 8 mdp mdp NOUN bracis-19024 137 9 we we PRON bracis-19024 137 10 begin begin VERB bracis-19024 137 11 by by ADP bracis-19024 137 12 following follow VERB bracis-19024 137 13 the the DET bracis-19024 137 14 design design NOUN bracis-19024 137 15 of of ADP bracis-19024 137 16 deeprm deeprm NOUN bracis-19024 137 17     SPACE bracis-19024 138 1 [ [ X bracis-19024 138 2 7 7 NUM bracis-19024 138 3 ] ] PUNCT bracis-19024 138 4 , , PUNCT bracis-19024 138 5 summarized summarize VERB bracis-19024 138 6 here here ADV bracis-19024 138 7 , , PUNCT bracis-19024 138 8 and and CCONJ bracis-19024 138 9 exemplified exemplify VERB bracis-19024 138 10 in in ADP bracis-19024 138 11 fig fig NOUN bracis-19024 138 12 . . PUNCT bracis-19024 138 13     SPACE bracis-19024 139 1 2 2 X bracis-19024 139 2 . . X bracis-19024 140 1 we we PRON bracis-19024 140 2 start start VERB bracis-19024 140 3 by by ADP bracis-19024 140 4 representing represent VERB bracis-19024 140 5 states state NOUN bracis-19024 140 6 as as ADP bracis-19024 140 7 images image NOUN bracis-19024 140 8 whose whose DET bracis-19024 140 9 height height NOUN bracis-19024 140 10 corresponds correspond VERB bracis-19024 140 11 to to ADP bracis-19024 140 12 a a DET bracis-19024 140 13 look look NOUN bracis-19024 140 14 to to ADP bracis-19024 140 15 a a DET bracis-19024 140 16 time time NOUN bracis-19024 140 17 horizon horizon NOUN bracis-19024 140 18 of of ADP bracis-19024 140 19 h h PROPN bracis-19024 140 20 time time NOUN bracis-19024 140 21 - - PUNCT bracis-19024 140 22 steps step NOUN bracis-19024 140 23 “ " PUNCT bracis-19024 140 24 into into ADP bracis-19024 140 25 ” " PUNCT bracis-19024 140 26 the the DET bracis-19024 140 27 future future NOUN bracis-19024 140 28 , , PUNCT bracis-19024 140 29 and and CCONJ bracis-19024 140 30 the the DET bracis-19024 140 31 width width ADJ bracis-19024 140 32 comprising comprising NOUN bracis-19024 140 33 : : PUNCT bracis-19024 140 34 the the DET bracis-19024 140 35 number number NOUN bracis-19024 140 36 of of ADP bracis-19024 140 37 processors processor NOUN bracis-19024 140 38 in in ADP bracis-19024 140 39 the the DET bracis-19024 140 40 system system NOUN bracis-19024 140 41 and and CCONJ bracis-19024 140 42 their their PRON bracis-19024 140 43 occupancy occupancy NOUN bracis-19024 140 44 state state NOUN bracis-19024 140 45 , , PUNCT bracis-19024 140 46 a a DET bracis-19024 140 47 window window NOUN bracis-19024 140 48 of of ADP bracis-19024 140 49 configurable configurable ADJ bracis-19024 140 50 size size NOUN bracis-19024 140 51 w w PROPN bracis-19024 140 52 ( ( PUNCT bracis-19024 140 53 in in ADP bracis-19024 140 54 the the DET bracis-19024 140 55 figure figure NOUN bracis-19024 140 56 , , PUNCT bracis-19024 140 57 \(w=2\ \(w=2\ NOUN bracis-19024 140 58 ) ) PUNCT bracis-19024 140 59 ) ) PUNCT bracis-19024 140 60 containing contain VERB bracis-19024 140 61 the the DET bracis-19024 140 62 first first ADJ bracis-19024 140 63 w w NOUN bracis-19024 140 64 jobs job NOUN bracis-19024 140 65 in in ADP bracis-19024 140 66 the the DET bracis-19024 140 67 wait wait NOUN bracis-19024 140 68 queue queue NOUN bracis-19024 140 69 times time NOUN bracis-19024 140 70 the the DET bracis-19024 140 71 number number NOUN bracis-19024 140 72 of of ADP bracis-19024 140 73 processors processor NOUN bracis-19024 140 74 in in ADP bracis-19024 140 75 the the DET bracis-19024 140 76 system system NOUN bracis-19024 140 77 , , PUNCT bracis-19024 140 78 and and CCONJ bracis-19024 140 79 a a DET bracis-19024 140 80 column column NOUN bracis-19024 140 81 vector vector NOUN bracis-19024 140 82 indicating indicate VERB bracis-19024 140 83 jobs job NOUN bracis-19024 140 84 in in ADP bracis-19024 140 85 a a DET bracis-19024 140 86 “ " PUNCT bracis-19024 140 87 backlog”footnote backlog”footnote NUM bracis-19024 140 88 6 6 NUM bracis-19024 140 89 . . PUNCT bracis-19024 141 1 if if SCONJ bracis-19024 141 2 there there PRON bracis-19024 141 3 are be VERB bracis-19024 141 4 more more ADJ bracis-19024 141 5 jobs job NOUN bracis-19024 141 6 in in ADP bracis-19024 141 7 the the DET bracis-19024 141 8 system system NOUN bracis-19024 141 9 that that PRON bracis-19024 141 10 can can AUX bracis-19024 141 11 fit fit VERB bracis-19024 141 12 the the DET bracis-19024 141 13 window window NOUN bracis-19024 141 14 and and CCONJ bracis-19024 141 15 the the DET bracis-19024 141 16 backlog backlog NOUN bracis-19024 141 17 , , PUNCT bracis-19024 141 18 they they PRON bracis-19024 141 19 are be AUX bracis-19024 141 20 omitted omit VERB bracis-19024 141 21 from from ADP bracis-19024 141 22 the the DET bracis-19024 141 23 state state NOUN bracis-19024 141 24 representationfootnote representationfootnote VERB bracis-19024 141 25 7 7 NUM bracis-19024 141 26 . . PUNCT bracis-19024 141 27 fig fig NOUN bracis-19024 141 28 . . PUNCT bracis-19024 142 1 2 2 X bracis-19024 142 2 . . X bracis-19024 142 3 dense dense ADJ bracis-19024 142 4 state state NOUN bracis-19024 142 5 representation representation NOUN bracis-19024 142 6 with with ADP bracis-19024 142 7 images image NOUN bracis-19024 142 8 representing represent VERB bracis-19024 142 9 state state NOUN bracis-19024 142 10 . . PUNCT bracis-19024 143 1 in in ADP bracis-19024 143 2 the the DET bracis-19024 143 3 figure figure NOUN bracis-19024 143 4 , , PUNCT bracis-19024 143 5 there there PRON bracis-19024 143 6 is be VERB bracis-19024 143 7 one one NUM bracis-19024 143 8 job job NOUN bracis-19024 143 9 in in ADP bracis-19024 143 10 execution execution NOUN bracis-19024 143 11 ( ( PUNCT bracis-19024 143 12 with with ADP bracis-19024 143 13 two two NUM bracis-19024 143 14 processors processor NOUN bracis-19024 143 15 for for ADP bracis-19024 143 16 the the DET bracis-19024 143 17 next next ADJ bracis-19024 143 18 two two NUM bracis-19024 143 19 time time NOUN bracis-19024 143 20 steps step NOUN bracis-19024 143 21 ) ) PUNCT bracis-19024 143 22 , , PUNCT bracis-19024 143 23 three three NUM bracis-19024 143 24 waiting wait VERB bracis-19024 143 25 jobs job NOUN bracis-19024 143 26 in in ADP bracis-19024 143 27     SPACE bracis-19024 143 28 total total NOUN bracis-19024 143 29 , , PUNCT bracis-19024 143 30 two two NUM bracis-19024 143 31 of of ADP bracis-19024 143 32 them they PRON bracis-19024 143 33 within within ADP bracis-19024 143 34 window window NOUN bracis-19024 143 35 \(w=2\ \(w=2\ PROPN bracis-19024 143 36 ) ) PUNCT bracis-19024 143 37 , , PUNCT bracis-19024 143 38 one one NOUN bracis-19024 143 39 using use VERB bracis-19024 143 40 one one NUM bracis-19024 143 41 cpu cpu NOUN bracis-19024 143 42 for for ADP bracis-19024 143 43 at at ADV bracis-19024 143 44 least least ADV bracis-19024 143 45 five five NUM bracis-19024 143 46 time time NOUN bracis-19024 143 47 steps step NOUN bracis-19024 143 48 , , PUNCT bracis-19024 143 49 and and CCONJ bracis-19024 143 50 another another PRON bracis-19024 143 51 using use VERB bracis-19024 143 52 three three NUM bracis-19024 143 53 cpus cpu NOUN bracis-19024 143 54 for for ADP bracis-19024 143 55 four four NUM bracis-19024 143 56 time time NOUN bracis-19024 143 57 steps step NOUN bracis-19024 143 58 . . PUNCT bracis-19024 144 1 details detail NOUN bracis-19024 144 2 for for ADP bracis-19024 144 3 the the DET bracis-19024 144 4 third third ADJ bracis-19024 144 5 job job NOUN bracis-19024 144 6 , , PUNCT bracis-19024 144 7 in in ADP bracis-19024 144 8 the the DET bracis-19024 144 9 backlog backlog NOUN bracis-19024 144 10 are be AUX bracis-19024 144 11 omitted omit VERB bracis-19024 144 12 . . PUNCT bracis-19024 145 1 full full ADJ bracis-19024 145 2 size size NOUN bracis-19024 145 3 image image NOUN bracis-19024 145 4 for for ADP bracis-19024 145 5 the the DET bracis-19024 145 6 action action NOUN bracis-19024 145 7 , , PUNCT bracis-19024 145 8 the the DET bracis-19024 145 9 agent agent NOUN bracis-19024 145 10 can can AUX bracis-19024 145 11 either either CCONJ bracis-19024 145 12 choose choose VERB bracis-19024 145 13 to to PART bracis-19024 145 14 schedule schedule VERB bracis-19024 145 15 a a DET bracis-19024 145 16 job job NOUN bracis-19024 145 17 from from ADP bracis-19024 145 18 one one NUM bracis-19024 145 19 of of ADP bracis-19024 145 20 the the DET bracis-19024 145 21 w w NOUN bracis-19024 145 22 slots slot NOUN bracis-19024 145 23 , , PUNCT bracis-19024 145 24 or or CCONJ bracis-19024 145 25 it it PRON bracis-19024 145 26 can can AUX bracis-19024 145 27 refuse refuse VERB bracis-19024 145 28 to to PART bracis-19024 145 29 schedule schedule VERB bracis-19024 145 30 a a DET bracis-19024 145 31 job job NOUN bracis-19024 145 32 , , PUNCT bracis-19024 145 33 totalling total VERB bracis-19024 145 34 three three NUM bracis-19024 145 35 possible possible ADJ bracis-19024 145 36 actions action NOUN bracis-19024 145 37 in in ADP bracis-19024 145 38 the the DET bracis-19024 145 39 example example NOUN bracis-19024 145 40 of of ADP bracis-19024 145 41 fig fig NOUN bracis-19024 145 42 . . PUNCT bracis-19024 145 43     SPACE bracis-19024 146 1 2 2 NUM bracis-19024 146 2 . . PUNCT bracis-19024 146 3 regarding regard VERB bracis-19024 146 4 when when SCONJ bracis-19024 146 5 actions action NOUN bracis-19024 146 6 are be AUX bracis-19024 146 7 taken take VERB bracis-19024 146 8 , , PUNCT bracis-19024 146 9 the the DET bracis-19024 146 10 mdp mdp NOUN bracis-19024 146 11 was be AUX bracis-19024 146 12 built build VERB bracis-19024 146 13 in in ADP bracis-19024 146 14 such such DET bracis-19024 146 15 a a DET bracis-19024 146 16 way way NOUN bracis-19024 146 17 that that PRON bracis-19024 146 18 agents agent NOUN bracis-19024 146 19 see see VERB bracis-19024 146 20 every every DET bracis-19024 146 21 simulation simulation NOUN bracis-19024 146 22 time time NOUN bracis-19024 146 23 step step NOUN bracis-19024 146 24 and and CCONJ bracis-19024 146 25 “ " PUNCT bracis-19024 146 26 intermediate intermediate ADJ bracis-19024 146 27 ” " PUNCT bracis-19024 146 28 time time NOUN bracis-19024 146 29 - - PUNCT bracis-19024 146 30 steps step NOUN bracis-19024 146 31 as as ADV bracis-19024 146 32 well well ADV bracis-19024 146 33 : : PUNCT bracis-19024 146 34 when when SCONJ bracis-19024 146 35 a a DET bracis-19024 146 36 job job NOUN bracis-19024 146 37 is be AUX bracis-19024 146 38 scheduled schedule VERB bracis-19024 146 39 , , PUNCT bracis-19024 146 40 there there PRON bracis-19024 146 41 is be VERB bracis-19024 146 42 a a DET bracis-19024 146 43 state state NOUN bracis-19024 146 44 change change NOUN bracis-19024 146 45 in in ADP bracis-19024 146 46 the the DET bracis-19024 146 47 mdp mdp NOUN bracis-19024 146 48 , , PUNCT bracis-19024 146 49 with with ADP bracis-19024 146 50 the the DET bracis-19024 146 51 job job NOUN bracis-19024 146 52 moving move VERB bracis-19024 146 53 to to ADP bracis-19024 146 54 the the DET bracis-19024 146 55 in in ADP bracis-19024 146 56 - - PUNCT bracis-19024 146 57 use use NOUN bracis-19024 146 58 processors processor NOUN bracis-19024 146 59 , , PUNCT bracis-19024 146 60 and and CCONJ bracis-19024 146 61 the the DET bracis-19024 146 62 queue queue NOUN bracis-19024 146 63 being be AUX bracis-19024 146 64 re re VERB bracis-19024 146 65 - - VERB bracis-19024 146 66 organized organize VERB bracis-19024 146 67 so so SCONJ bracis-19024 146 68 that that SCONJ bracis-19024 146 69 all all DET bracis-19024 146 70 slots slot NOUN bracis-19024 146 71 in in ADP bracis-19024 146 72 window window NOUN bracis-19024 146 73 w w NOUN bracis-19024 146 74 are be AUX bracis-19024 146 75 filled fill VERB bracis-19024 146 76 . . PUNCT bracis-19024 147 1 in in ADP bracis-19024 147 2 the the DET bracis-19024 147 3 base base NOUN bracis-19024 147 4 mdp mdp PROPN bracis-19024 147 5 , , PUNCT bracis-19024 147 6 whenever whenever SCONJ bracis-19024 147 7 a a DET bracis-19024 147 8 job job NOUN bracis-19024 147 9 is be AUX bracis-19024 147 10 scheduled schedule VERB bracis-19024 147 11 , , PUNCT bracis-19024 147 12 the the DET bracis-19024 147 13 agent agent NOUN bracis-19024 147 14 receives receive VERB bracis-19024 147 15 a a DET bracis-19024 147 16 reward reward NOUN bracis-19024 147 17 of of ADP bracis-19024 147 18 zero zero NUM bracis-19024 147 19 . . PUNCT bracis-19024 148 1 in in ADP bracis-19024 148 2 all all DET bracis-19024 148 3 other other ADJ bracis-19024 148 4 cases case NOUN bracis-19024 148 5 , , PUNCT bracis-19024 148 6 the the DET bracis-19024 148 7 reward reward NOUN bracis-19024 148 8 is be AUX bracis-19024 148 9 given give VERB bracis-19024 148 10 by by ADP bracis-19024 148 11 the the DET bracis-19024 148 12 negative negative ADJ bracis-19024 148 13 online online ADJ bracis-19024 148 14 slowdown slowdown NOUN bracis-19024 148 15     SPACE bracis-19024 148 16 ( ( PUNCT bracis-19024 148 17 1 1 NUM bracis-19024 148 18 ): ): PUNCT bracis-19024 148 19 \(\mathcal \(\mathcal ADJ bracis-19024 148 20 { { PUNCT bracis-19024 148 21 r}=-\sum r}=-\sum PROPN bracis-19024 148 22 _ _ PROPN bracis-19024 148 23 { { PUNCT bracis-19024 148 24 j\in j\in PROPN bracis-19024 148 25 \mathcal \mathcal PROPN bracis-19024 148 26 { { PUNCT bracis-19024 148 27 j}}\frac{1}{t_e(j)}\ j}}\frac{1}{t_e(j)}\ PROPN bracis-19024 148 28 ) ) PUNCT bracis-19024 148 29 . . PUNCT bracis-19024 149 1 for for ADP bracis-19024 149 2 a a DET bracis-19024 149 3 detailed detailed ADJ bracis-19024 149 4 description description NOUN bracis-19024 149 5 of of ADP bracis-19024 149 6 the the DET bracis-19024 149 7 environment environment NOUN bracis-19024 149 8 , , PUNCT bracis-19024 149 9 we we PRON bracis-19024 149 10 direct direct VERB bracis-19024 149 11 the the DET bracis-19024 149 12 reader reader NOUN bracis-19024 149 13 to to ADP bracis-19024 149 14     SPACE bracis-19024 149 15 mao mao PROPN bracis-19024 149 16 et et PROPN bracis-19024 149 17 al al PROPN bracis-19024 149 18 . . PUNCT bracis-19024 150 1 [ [ X bracis-19024 150 2 7 7 NUM bracis-19024 150 3 ] ] PUNCT bracis-19024 150 4 , , PUNCT bracis-19024 150 5 who who PRON bracis-19024 150 6 first first ADV bracis-19024 150 7 introduced introduce VERB bracis-19024 150 8 it it PRON bracis-19024 150 9 . . PUNCT bracis-19024 151 1 4.2 4.2 NUM bracis-19024 151 2 compact compact ADJ bracis-19024 151 3 state state NOUN bracis-19024 151 4 representation representation NOUN bracis-19024 151 5 the the DET bracis-19024 151 6 first first ADJ bracis-19024 151 7 realization realization NOUN bracis-19024 151 8 we we PRON bracis-19024 151 9 had have VERB bracis-19024 151 10 was be AUX bracis-19024 151 11 that that SCONJ bracis-19024 151 12 the the DET bracis-19024 151 13 state state NOUN bracis-19024 151 14 representation representation NOUN bracis-19024 151 15 in in ADP bracis-19024 151 16 the the DET bracis-19024 151 17 base base NOUN bracis-19024 151 18 mdp mdp NOUN bracis-19024 151 19 is be AUX bracis-19024 151 20 wasteful wasteful ADJ bracis-19024 151 21 , , PUNCT bracis-19024 151 22 in in ADP bracis-19024 151 23 the the DET bracis-19024 151 24 sense sense NOUN bracis-19024 151 25 that that SCONJ bracis-19024 151 26 one one PRON bracis-19024 151 27 can can AUX bracis-19024 151 28 reduce reduce VERB bracis-19024 151 29 the the DET bracis-19024 151 30 size size NOUN bracis-19024 151 31 of of ADP bracis-19024 151 32 the the DET bracis-19024 151 33 state state NOUN bracis-19024 151 34 without without ADP bracis-19024 151 35 losing lose VERB bracis-19024 151 36 information information NOUN bracis-19024 151 37 . . PUNCT bracis-19024 152 1 particularly particularly ADV bracis-19024 152 2 when when SCONJ bracis-19024 152 3 working work VERB bracis-19024 152 4 with with ADP bracis-19024 152 5 larger large ADJ bracis-19024 152 6 clusters cluster NOUN bracis-19024 152 7 , , PUNCT bracis-19024 152 8 or or CCONJ bracis-19024 152 9 with with ADP bracis-19024 152 10 a a DET bracis-19024 152 11 larger large ADJ bracis-19024 152 12 number number NOUN bracis-19024 152 13 of of ADP bracis-19024 152 14 job job NOUN bracis-19024 152 15 slots slot NOUN bracis-19024 152 16 , , PUNCT bracis-19024 152 17 it it PRON bracis-19024 152 18 may may AUX bracis-19024 152 19 be be AUX bracis-19024 152 20 the the DET bracis-19024 152 21 case case NOUN bracis-19024 152 22 that that SCONJ bracis-19024 152 23 full full ADJ bracis-19024 152 24 trajectories trajectory NOUN bracis-19024 152 25 take take VERB bracis-19024 152 26 too too ADV bracis-19024 152 27 much much ADJ bracis-19024 152 28 space space NOUN bracis-19024 152 29 in in ADP bracis-19024 152 30 memory memory NOUN bracis-19024 152 31 , , PUNCT bracis-19024 152 32 reducing reduce VERB bracis-19024 152 33 the the DET bracis-19024 152 34 computational computational ADJ bracis-19024 152 35 performance performance NOUN bracis-19024 152 36 of of ADP bracis-19024 152 37 learning learn VERB bracis-19024 152 38 agents agent NOUN bracis-19024 152 39 . . PUNCT bracis-19024 153 1 due due ADP bracis-19024 153 2 to to ADP bracis-19024 153 3 that that PRON bracis-19024 153 4 , , PUNCT bracis-19024 153 5 and and CCONJ bracis-19024 153 6 based base VERB bracis-19024 153 7 on on ADP bracis-19024 153 8 a a DET bracis-19024 153 9 set set NOUN bracis-19024 153 10 of of ADP bracis-19024 153 11 features feature NOUN bracis-19024 153 12 found find VERB bracis-19024 153 13 in in ADP bracis-19024 153 14 the the DET bracis-19024 153 15 literature literature NOUN bracis-19024 153 16 for for ADP bracis-19024 153 17 machine machine NOUN bracis-19024 153 18 learning learn VERB bracis-19024 153 19 with with ADP bracis-19024 153 20 hpc hpc PROPN bracis-19024 153 21 jobs job NOUN bracis-19024 153 22     SPACE bracis-19024 153 23 [ [ X bracis-19024 153 24 1 1 NUM bracis-19024 153 25 ] ] PUNCT bracis-19024 153 26 , , PUNCT bracis-19024 153 27 we we PRON bracis-19024 153 28 devised devise VERB bracis-19024 153 29 a a DET bracis-19024 153 30 set set NOUN bracis-19024 153 31 of of ADP bracis-19024 153 32 features feature NOUN bracis-19024 153 33 that that PRON bracis-19024 153 34 can can AUX bracis-19024 153 35 represent represent VERB bracis-19024 153 36 states state NOUN bracis-19024 153 37 in in ADP bracis-19024 153 38 a a DET bracis-19024 153 39 compact compact ADJ bracis-19024 153 40 way way NOUN bracis-19024 153 41 . . PUNCT bracis-19024 154 1 in in ADP bracis-19024 154 2 our our PRON bracis-19024 154 3 new new ADJ bracis-19024 154 4 state state NOUN bracis-19024 154 5 representation representation NOUN bracis-19024 154 6 , , PUNCT bracis-19024 154 7 jobs job NOUN bracis-19024 154 8 in in ADP bracis-19024 154 9 the the DET bracis-19024 154 10 queue queue NOUN bracis-19024 154 11 are be AUX bracis-19024 154 12 represented represent VERB bracis-19024 154 13 by by ADP bracis-19024 154 14 the the DET bracis-19024 154 15 features feature NOUN bracis-19024 154 16 shown show VERB bracis-19024 154 17 in in ADP bracis-19024 154 18 table table NOUN bracis-19024 154 19     SPACE bracis-19024 154 20 1 1 NUM bracis-19024 154 21 , , PUNCT bracis-19024 154 22 where where SCONJ bracis-19024 154 23 “ " PUNCT bracis-19024 154 24 work work NOUN bracis-19024 154 25 ” " PUNCT bracis-19024 154 26 is be AUX bracis-19024 154 27 computed compute VERB bracis-19024 154 28 by by ADP bracis-19024 154 29 multiplying multiply VERB bracis-19024 154 30 the the DET bracis-19024 154 31 number number NOUN bracis-19024 154 32 of of ADP bracis-19024 154 33 processors processor NOUN bracis-19024 154 34 a a DET bracis-19024 154 35 job job NOUN bracis-19024 154 36 requires require VERB bracis-19024 154 37 by by ADP bracis-19024 154 38 the the DET bracis-19024 154 39 time time NOUN bracis-19024 154 40 it it PRON bracis-19024 154 41 is be AUX bracis-19024 154 42 expected expect VERB bracis-19024 154 43 to to PART bracis-19024 154 44 run run VERB bracis-19024 154 45 , , PUNCT bracis-19024 154 46 and and CCONJ bracis-19024 154 47 cluster cluster NOUN bracis-19024 154 48 features feature NOUN bracis-19024 154 49 are be AUX bracis-19024 154 50 a a DET bracis-19024 154 51 pair pair NOUN bracis-19024 154 52 that that PRON bracis-19024 154 53 indicates indicate VERB bracis-19024 154 54 the the DET bracis-19024 154 55 number number NOUN bracis-19024 154 56 of of ADP bracis-19024 154 57 processors processor NOUN bracis-19024 154 58 in in ADP bracis-19024 154 59 use use NOUN bracis-19024 154 60 , , PUNCT bracis-19024 154 61 and and CCONJ bracis-19024 154 62 the the DET bracis-19024 154 63 number number NOUN bracis-19024 154 64 of of ADP bracis-19024 154 65 free free ADJ bracis-19024 154 66 processors processor NOUN bracis-19024 154 67 . . PUNCT bracis-19024 155 1 the the DET bracis-19024 155 2 features feature NOUN bracis-19024 155 3 related relate VERB bracis-19024 155 4 to to ADP bracis-19024 155 5 the the DET bracis-19024 155 6 cluster cluster NOUN bracis-19024 155 7 state state NOUN bracis-19024 155 8 still still ADV bracis-19024 155 9 use use VERB bracis-19024 155 10 a a DET bracis-19024 155 11 time time NOUN bracis-19024 155 12 horizon horizon NOUN bracis-19024 155 13 h h NOUN bracis-19024 155 14 but but CCONJ bracis-19024 155 15 instead instead ADV bracis-19024 155 16 of of ADP bracis-19024 155 17 using use VERB bracis-19024 155 18 a a DET bracis-19024 155 19 matrix matrix NOUN bracis-19024 155 20 , , PUNCT bracis-19024 155 21 we we PRON bracis-19024 155 22 used use VERB bracis-19024 155 23 a a DET bracis-19024 155 24 pair pair NOUN bracis-19024 155 25 of of ADP bracis-19024 155 26 integers integer NOUN bracis-19024 155 27 representing represent VERB bracis-19024 155 28 how how SCONJ bracis-19024 155 29 many many ADJ bracis-19024 155 30 processors processor NOUN bracis-19024 155 31 are be AUX bracis-19024 155 32 in in ADP bracis-19024 155 33 use use NOUN bracis-19024 155 34 , , PUNCT bracis-19024 155 35 and and CCONJ bracis-19024 155 36 how how SCONJ bracis-19024 155 37 many many ADJ bracis-19024 155 38 processors processor NOUN bracis-19024 155 39 are be AUX bracis-19024 155 40 free free ADJ bracis-19024 155 41 in in ADP bracis-19024 155 42 a a DET bracis-19024 155 43 given give VERB bracis-19024 155 44 time time NOUN bracis-19024 155 45 - - PUNCT bracis-19024 155 46 step step NOUN bracis-19024 155 47 . . PUNCT bracis-19024 156 1 as as ADP bracis-19024 156 2 an an DET bracis-19024 156 3 example example NOUN bracis-19024 156 4 , , PUNCT bracis-19024 156 5 assuming assume VERB bracis-19024 156 6 the the DET bracis-19024 156 7 job job NOUN bracis-19024 156 8 in in ADP bracis-19024 156 9 the the DET bracis-19024 156 10 cluster cluster NOUN bracis-19024 156 11 was be AUX bracis-19024 156 12 submitted submit VERB bracis-19024 156 13 at at ADP bracis-19024 156 14 time time NOUN bracis-19024 156 15 1 1 NUM bracis-19024 156 16 , , PUNCT bracis-19024 156 17 the the DET bracis-19024 156 18 job job NOUN bracis-19024 156 19 in in ADP bracis-19024 156 20 slot slot NOUN bracis-19024 156 21 1 1 NUM bracis-19024 156 22 was be AUX bracis-19024 156 23 submitted submit VERB bracis-19024 156 24 at at ADP bracis-19024 156 25 time time NOUN bracis-19024 156 26 2 2 NUM bracis-19024 156 27 , , PUNCT bracis-19024 156 28 and and CCONJ bracis-19024 156 29 the the DET bracis-19024 156 30 job job NOUN bracis-19024 156 31 in in ADP bracis-19024 156 32 slot slot NOUN bracis-19024 156 33 2 2 NUM bracis-19024 156 34 was be AUX bracis-19024 156 35 submitted submit VERB bracis-19024 156 36 in in ADP bracis-19024 156 37 time time NOUN bracis-19024 156 38 3 3 NUM bracis-19024 156 39 , , PUNCT bracis-19024 156 40 the the DET bracis-19024 156 41 state state NOUN bracis-19024 156 42 shown show VERB bracis-19024 156 43 in in ADP bracis-19024 156 44 fig fig NOUN bracis-19024 156 45 . . PUNCT bracis-19024 156 46     SPACE bracis-19024 157 1 2 2 NUM bracis-19024 157 2 can can AUX bracis-19024 157 3 be be AUX bracis-19024 157 4 fully fully ADV bracis-19024 157 5 described describe VERB bracis-19024 157 6 by by ADP bracis-19024 157 7 the the DET bracis-19024 157 8 concatenation concatenation NOUN bracis-19024 157 9 of of ADP bracis-19024 157 10 vectors vector NOUN bracis-19024 157 11 with with ADP bracis-19024 157 12 cluster cluster NOUN bracis-19024 157 13 state state NOUN bracis-19024 157 14 \(\langle \(\langle PROPN bracis-19024 157 15 ( ( PUNCT bracis-19024 157 16 2 2 NUM bracis-19024 157 17 , , PUNCT bracis-19024 157 18 1 1 NUM bracis-19024 157 19 ) ) PUNCT bracis-19024 157 20 , , PUNCT bracis-19024 157 21 ( ( PUNCT bracis-19024 157 22 2 2 NUM bracis-19024 157 23 , , PUNCT bracis-19024 157 24 1 1 NUM bracis-19024 157 25 ) ) PUNCT bracis-19024 157 26 , , PUNCT bracis-19024 157 27 ( ( PUNCT bracis-19024 157 28 0 0 NUM bracis-19024 157 29 , , PUNCT bracis-19024 157 30 3 3 NUM bracis-19024 157 31 ) ) PUNCT bracis-19024 157 32 , , PUNCT bracis-19024 157 33 ( ( PUNCT bracis-19024 157 34 0 0 NUM bracis-19024 157 35 , , PUNCT bracis-19024 157 36 3 3 NUM bracis-19024 157 37 ) ) PUNCT bracis-19024 157 38 , , PUNCT bracis-19024 157 39 ( ( PUNCT bracis-19024 157 40 0 0 NUM bracis-19024 157 41 , , PUNCT bracis-19024 157 42 3 3 X bracis-19024 157 43 ) ) PUNCT bracis-19024 157 44 \rangle \rangle PROPN bracis-19024 157 45 \ \ NOUN bracis-19024 157 46 ) ) PUNCT bracis-19024 157 47 , , PUNCT bracis-19024 157 48 jobs job NOUN bracis-19024 157 49 in in ADP bracis-19024 157 50 window window NOUN bracis-19024 157 51 w w PROPN bracis-19024 157 52 \(\langle \(\langle PROPN bracis-19024 157 53 ( ( PUNCT bracis-19024 157 54 1 1 NUM bracis-19024 157 55 , , PUNCT bracis-19024 157 56 5 5 NUM bracis-19024 157 57 , , PUNCT bracis-19024 157 58 1 1 NUM bracis-19024 157 59 , , PUNCT bracis-19024 157 60 0 0 NUM bracis-19024 157 61 , , PUNCT bracis-19024 157 62 0 0 NUM bracis-19024 157 63 , , PUNCT bracis-19024 157 64 1 1 NUM bracis-19024 157 65 , , PUNCT bracis-19024 157 66 6 6 NUM bracis-19024 157 67 , , PUNCT bracis-19024 157 68 0 0 NUM bracis-19024 157 69 ) ) PUNCT bracis-19024 157 70 , , PUNCT bracis-19024 157 71 ( ( PUNCT bracis-19024 157 72 2 2 NUM bracis-19024 157 73 , , PUNCT bracis-19024 157 74 4 4 NUM bracis-19024 157 75 , , PUNCT bracis-19024 157 76 3 3 NUM bracis-19024 157 77 , , PUNCT bracis-19024 157 78 1 1 NUM bracis-19024 157 79 , , PUNCT bracis-19024 157 80 5 5 NUM bracis-19024 157 81 , , PUNCT bracis-19024 157 82 1 1 NUM bracis-19024 157 83 , , PUNCT bracis-19024 157 84 4 4 NUM bracis-19024 157 85 , , PUNCT bracis-19024 157 86 0 0 NUM bracis-19024 157 87 ) ) PUNCT bracis-19024 157 88 \rangle \rangle PROPN bracis-19024 157 89 \ \ NOUN bracis-19024 157 90 ) ) PUNCT bracis-19024 157 91 and and CCONJ bracis-19024 157 92 backlog backlog NOUN bracis-19024 157 93 \(\langle \(\langle PROPN bracis-19024 157 94 1 1 NUM bracis-19024 157 95 \rangle \rangle NOUN bracis-19024 157 96 \)footnote \)footnote NUM bracis-19024 157 97 8 8 NUM bracis-19024 157 98 . . PUNCT bracis-19024 158 1 the the DET bracis-19024 158 2 features feature NOUN bracis-19024 158 3 in in ADP bracis-19024 158 4 the the DET bracis-19024 158 5 jobs job NOUN bracis-19024 158 6 slots slot NOUN bracis-19024 158 7 are be AUX bracis-19024 158 8 presented present VERB bracis-19024 158 9 in in ADP bracis-19024 158 10 the the DET bracis-19024 158 11 same same ADJ bracis-19024 158 12 order order NOUN bracis-19024 158 13 as as ADP bracis-19024 158 14 the the DET bracis-19024 158 15 ones one NOUN bracis-19024 158 16 shown show VERB bracis-19024 158 17 in in ADP bracis-19024 158 18 table table NOUN bracis-19024 158 19     SPACE bracis-19024 158 20 1 1 NUM bracis-19024 158 21 . . PUNCT bracis-19024 158 22 table table NOUN bracis-19024 158 23 1 1 NUM bracis-19024 158 24 . . PUNCT bracis-19024 158 25 job job NOUN bracis-19024 158 26 features feature NOUN bracis-19024 158 27 in in ADP bracis-19024 158 28 a a DET bracis-19024 158 29 compact compact ADJ bracis-19024 158 30 state state NOUN bracis-19024 158 31 representation.full representation.full ADJ bracis-19024 158 32 size size NOUN bracis-19024 158 33 table table NOUN bracis-19024 158 34 a a DET bracis-19024 158 35 side side NOUN bracis-19024 158 36 - - PUNCT bracis-19024 158 37 effect effect NOUN bracis-19024 158 38 of of ADP bracis-19024 158 39 using use VERB bracis-19024 158 40 this this DET bracis-19024 158 41 new new ADJ bracis-19024 158 42 compact compact ADJ bracis-19024 158 43 state state NOUN bracis-19024 158 44 representation representation NOUN bracis-19024 158 45 is be AUX bracis-19024 158 46 that that SCONJ bracis-19024 158 47 , , PUNCT bracis-19024 158 48 when when SCONJ bracis-19024 158 49 h h NOUN bracis-19024 158 50 and and CCONJ bracis-19024 158 51 w w PROPN bracis-19024 158 52 are be AUX bracis-19024 158 53 fixed fix VERB bracis-19024 158 54 between between ADP bracis-19024 158 55 different different ADJ bracis-19024 158 56 cluster cluster NOUN bracis-19024 158 57 configurations configuration NOUN bracis-19024 158 58 , , PUNCT bracis-19024 158 59 learned learn VERB bracis-19024 158 60 features feature NOUN bracis-19024 158 61 are be AUX bracis-19024 158 62 directly directly ADV bracis-19024 158 63 transferable transferable ADJ bracis-19024 158 64 between between ADP bracis-19024 158 65 clusters cluster NOUN bracis-19024 158 66 even even ADV bracis-19024 158 67 when when SCONJ bracis-19024 158 68 using use VERB bracis-19024 158 69 function function NOUN bracis-19024 158 70 approximation approximation NOUN bracis-19024 158 71 methods method NOUN bracis-19024 158 72 that that PRON bracis-19024 158 73 depend depend VERB bracis-19024 158 74 on on ADP bracis-19024 158 75 a a DET bracis-19024 158 76 fixed fix VERB bracis-19024 158 77 number number NOUN bracis-19024 158 78 of of ADP bracis-19024 158 79 features feature NOUN bracis-19024 158 80 . . PUNCT bracis-19024 159 1 4.3 4.3 NUM bracis-19024 159 2 sparse sparse ADJ bracis-19024 159 3 state state NOUN bracis-19024 159 4 transitions transition NOUN bracis-19024 159 5 another another DET bracis-19024 159 6 deficiency deficiency NOUN bracis-19024 159 7 we we PRON bracis-19024 159 8 ’ve ’ve AUX bracis-19024 159 9 identified identify VERB bracis-19024 159 10 in in ADP bracis-19024 159 11 the the DET bracis-19024 159 12 base base NOUN bracis-19024 159 13 mdp mdp NOUN bracis-19024 159 14 is be AUX bracis-19024 159 15 that that SCONJ bracis-19024 159 16 the the DET bracis-19024 159 17 agent agent NOUN bracis-19024 159 18 sees see VERB bracis-19024 159 19 all all DET bracis-19024 159 20 time time NOUN bracis-19024 159 21 - - PUNCT bracis-19024 159 22 steps step NOUN bracis-19024 159 23 in in ADP bracis-19024 159 24 the the DET bracis-19024 159 25 simulation simulation NOUN bracis-19024 159 26 , , PUNCT bracis-19024 159 27 but but CCONJ bracis-19024 159 28 this this PRON bracis-19024 159 29 causes cause VERB bracis-19024 159 30 the the DET bracis-19024 159 31 agent agent NOUN bracis-19024 159 32 to to PART bracis-19024 159 33 have have VERB bracis-19024 159 34 to to PART bracis-19024 159 35 take take VERB bracis-19024 159 36 an an DET bracis-19024 159 37 action action NOUN bracis-19024 159 38 even even ADV bracis-19024 159 39 when when SCONJ bracis-19024 159 40 there there PRON bracis-19024 159 41 is be VERB bracis-19024 159 42 no no DET bracis-19024 159 43 good good ADJ bracis-19024 159 44 action action NOUN bracis-19024 159 45 to to PART bracis-19024 159 46 take take VERB bracis-19024 159 47 . . PUNCT bracis-19024 160 1 consider consider VERB bracis-19024 160 2 , , PUNCT bracis-19024 160 3 for for ADP bracis-19024 160 4 example example NOUN bracis-19024 160 5 , , PUNCT bracis-19024 160 6 the the DET bracis-19024 160 7 case case NOUN bracis-19024 160 8 in in ADP bracis-19024 160 9 which which PRON bracis-19024 160 10 all all DET bracis-19024 160 11 resources resource NOUN bracis-19024 160 12 are be AUX bracis-19024 160 13 in in ADP bracis-19024 160 14 use use NOUN bracis-19024 160 15 ( ( PUNCT bracis-19024 160 16 there there PRON bracis-19024 160 17 are be VERB bracis-19024 160 18 no no DET bracis-19024 160 19 free free ADJ bracis-19024 160 20 resources resource NOUN bracis-19024 160 21 ) ) PUNCT bracis-19024 160 22 . . PUNCT bracis-19024 161 1 in in ADP bracis-19024 161 2 cases case NOUN bracis-19024 161 3 such such ADJ bracis-19024 161 4 as as ADP bracis-19024 161 5 this this PRON bracis-19024 161 6 , , PUNCT bracis-19024 161 7 any any DET bracis-19024 161 8 action action NOUN bracis-19024 161 9 the the DET bracis-19024 161 10 agent agent NOUN bracis-19024 161 11 takes take VERB bracis-19024 161 12 will will AUX bracis-19024 161 13 lead lead VERB bracis-19024 161 14 to to ADP bracis-19024 161 15 the the DET bracis-19024 161 16 same same ADJ bracis-19024 161 17 outcome outcome NOUN bracis-19024 161 18 : : PUNCT bracis-19024 161 19 increasing increase VERB bracis-19024 161 20 the the DET bracis-19024 161 21 simulation simulation NOUN bracis-19024 161 22 clock clock NOUN bracis-19024 161 23 , , PUNCT bracis-19024 161 24 receiving receive VERB bracis-19024 161 25 negative negative ADJ bracis-19024 161 26 rewards reward NOUN bracis-19024 161 27 related relate VERB bracis-19024 161 28 to to ADP bracis-19024 161 29 the the DET bracis-19024 161 30 slowdown slowdown NOUN bracis-19024 161 31 of of ADP bracis-19024 161 32 the the DET bracis-19024 161 33 jobs job NOUN bracis-19024 161 34 , , PUNCT bracis-19024 161 35 and and CCONJ bracis-19024 161 36 having have VERB bracis-19024 161 37 no no DET bracis-19024 161 38 new new ADJ bracis-19024 161 39 jobs job NOUN bracis-19024 161 40 scheduled schedule VERB bracis-19024 161 41 . . PUNCT bracis-19024 162 1 this this PRON bracis-19024 162 2 will will AUX bracis-19024 162 3 be be AUX bracis-19024 162 4 repeated repeat VERB bracis-19024 162 5 for for ADP bracis-19024 162 6 all all DET bracis-19024 162 7 time time NOUN bracis-19024 162 8 steps step NOUN bracis-19024 162 9 between between ADP bracis-19024 162 10 the the DET bracis-19024 162 11 start start NOUN bracis-19024 162 12 of of ADP bracis-19024 162 13 the the DET bracis-19024 162 14 last last ADJ bracis-19024 162 15 job job NOUN bracis-19024 162 16 that that PRON bracis-19024 162 17 exhausted exhaust VERB bracis-19024 162 18 resources resource NOUN bracis-19024 162 19 until until ADP bracis-19024 162 20 the the DET bracis-19024 162 21 finish finish NOUN bracis-19024 162 22 of of ADP bracis-19024 162 23 the the DET bracis-19024 162 24 first first ADJ bracis-19024 162 25 job job NOUN bracis-19024 162 26 that that PRON bracis-19024 162 27 frees free VERB bracis-19024 162 28 them they PRON bracis-19024 162 29 , , PUNCT bracis-19024 162 30 causing cause VERB bracis-19024 162 31 non non ADJ bracis-19024 162 32 - - ADJ bracis-19024 162 33 negative negative ADJ bracis-19024 162 34 rewards reward NOUN bracis-19024 162 35 to to PART bracis-19024 162 36 be be AUX bracis-19024 162 37 more more ADV bracis-19024 162 38 sparse sparse ADJ bracis-19024 162 39 , , PUNCT bracis-19024 162 40 making make VERB bracis-19024 162 41 the the DET bracis-19024 162 42 reinforcement reinforcement NOUN bracis-19024 162 43 signal signal NOUN bracis-19024 162 44 noisier noisy ADJ bracis-19024 162 45 and and CCONJ bracis-19024 162 46 , , PUNCT bracis-19024 162 47 therefore therefore ADV bracis-19024 162 48 , , PUNCT bracis-19024 162 49 harder hard ADV bracis-19024 162 50 to to PART bracis-19024 162 51 learn learn VERB bracis-19024 162 52 . . PUNCT bracis-19024 163 1 the the DET bracis-19024 163 2 opposite opposite NOUN bracis-19024 163 3 is be AUX bracis-19024 163 4 also also ADV bracis-19024 163 5 true true ADJ bracis-19024 163 6 : : PUNCT bracis-19024 163 7 if if SCONJ bracis-19024 163 8 there there PRON bracis-19024 163 9 are be VERB bracis-19024 163 10 no no DET bracis-19024 163 11 jobs job NOUN bracis-19024 163 12 waiting wait VERB bracis-19024 163 13 to to PART bracis-19024 163 14 be be AUX bracis-19024 163 15 scheduled schedule VERB bracis-19024 163 16 , , PUNCT bracis-19024 163 17 no no ADV bracis-19024 163 18 matter matter ADV bracis-19024 163 19 what what PRON bracis-19024 163 20 the the DET bracis-19024 163 21 agent agent NOUN bracis-19024 163 22 chooses choose VERB bracis-19024 163 23 , , PUNCT bracis-19024 163 24 the the DET bracis-19024 163 25 outcome outcome NOUN bracis-19024 163 26 will will AUX bracis-19024 163 27 be be AUX bracis-19024 163 28 the the DET bracis-19024 163 29 same same ADJ bracis-19024 163 30 : : PUNCT bracis-19024 163 31 no no DET bracis-19024 163 32 jobs job NOUN bracis-19024 163 33 will will AUX bracis-19024 163 34 be be AUX bracis-19024 163 35 scheduled schedule VERB bracis-19024 163 36 . . PUNCT bracis-19024 164 1 due due ADP bracis-19024 164 2 to to ADP bracis-19024 164 3 that that PRON bracis-19024 164 4 , , PUNCT bracis-19024 164 5 we we PRON bracis-19024 164 6 updated update VERB bracis-19024 164 7 the the DET bracis-19024 164 8 environment environment NOUN bracis-19024 164 9 to to PART bracis-19024 164 10 only only ADV bracis-19024 164 11 call call VERB bracis-19024 164 12 the the DET bracis-19024 164 13 agent agent NOUN bracis-19024 164 14 and and CCONJ bracis-19024 164 15 , , PUNCT bracis-19024 164 16 therefore therefore ADV bracis-19024 164 17 , , PUNCT bracis-19024 164 18 to to PART bracis-19024 164 19 only only ADV bracis-19024 164 20 add add VERB bracis-19024 164 21 states state NOUN bracis-19024 164 22 , , PUNCT bracis-19024 164 23 actions action NOUN bracis-19024 164 24 and and CCONJ bracis-19024 164 25 rewards reward NOUN bracis-19024 164 26 to to ADP bracis-19024 164 27 a a DET bracis-19024 164 28 trajectory trajectory NOUN bracis-19024 164 29 , , PUNCT bracis-19024 164 30 when when SCONJ bracis-19024 164 31 it it PRON bracis-19024 164 32 was be AUX bracis-19024 164 33 possible possible ADJ bracis-19024 164 34 for for SCONJ bracis-19024 164 35 the the DET bracis-19024 164 36 agent agent NOUN bracis-19024 164 37 to to PART bracis-19024 164 38 take take VERB bracis-19024 164 39 an an DET bracis-19024 164 40 action action NOUN bracis-19024 164 41 that that PRON bracis-19024 164 42 could could AUX bracis-19024 164 43 result result VERB bracis-19024 164 44 in in ADP bracis-19024 164 45 a a DET bracis-19024 164 46 job job NOUN bracis-19024 164 47 being be AUX bracis-19024 164 48 scheduled schedule VERB bracis-19024 164 49 . . PUNCT bracis-19024 165 1 in in ADP bracis-19024 165 2 short short ADJ bracis-19024 165 3 , , PUNCT bracis-19024 165 4 we we PRON bracis-19024 165 5 change change VERB bracis-19024 165 6 the the DET bracis-19024 165 7 transition transition NOUN bracis-19024 165 8 function function NOUN bracis-19024 165 9 \(\mathcal \(\mathcal ADJ bracis-19024 165 10 { { PUNCT bracis-19024 165 11 t}(s_{t+1}\mid t}(s_{t+1}\mid NOUN bracis-19024 165 12 s_t s_t NOUN bracis-19024 165 13 , , PUNCT bracis-19024 165 14 a_t)\ a_t)\ NOUN bracis-19024 165 15 ) ) PUNCT bracis-19024 165 16 so so SCONJ bracis-19024 165 17 that that SCONJ bracis-19024 165 18 all all DET bracis-19024 165 19 state state NOUN bracis-19024 165 20 transitions transition NOUN bracis-19024 165 21 from from ADP bracis-19024 165 22 \(s_t\ \(s_t\ ADJ bracis-19024 165 23 ) ) PUNCT bracis-19024 165 24 to to ADP bracis-19024 165 25 \(s_{t+1}\ \(s_{t+1}\ NUM bracis-19024 165 26 ) ) PUNCT bracis-19024 165 27 will will AUX bracis-19024 165 28 always always ADV bracis-19024 165 29 have have VERB bracis-19024 165 30 at at ADV bracis-19024 165 31 least least ADV bracis-19024 165 32 one one NUM bracis-19024 165 33 job job NOUN bracis-19024 165 34 that that PRON bracis-19024 165 35 may may AUX bracis-19024 165 36 be be AUX bracis-19024 165 37 scheduled schedule VERB bracis-19024 165 38 by by ADP bracis-19024 165 39 the the DET bracis-19024 165 40 agent agent NOUN bracis-19024 165 41 in in ADP bracis-19024 165 42 state state NOUN bracis-19024 165 43 \(s_{t+1}\ \(s_{t+1}\ NUM bracis-19024 165 44 ) ) PUNCT bracis-19024 165 45 . . PUNCT bracis-19024 166 1 we we PRON bracis-19024 166 2 did do AUX bracis-19024 166 3 not not PART bracis-19024 166 4 change change VERB bracis-19024 166 5 the the DET bracis-19024 166 6 initial initial ADJ bracis-19024 166 7 state state NOUN bracis-19024 166 8 , , PUNCT bracis-19024 166 9 though though ADV bracis-19024 166 10 , , PUNCT bracis-19024 166 11 so so ADV bracis-19024 166 12 still still ADV bracis-19024 166 13 holds hold VERB bracis-19024 166 14 . . PUNCT bracis-19024 167 1 this this PRON bracis-19024 167 2 essentially essentially ADV bracis-19024 167 3 turns turn VERB bracis-19024 167 4 the the DET bracis-19024 167 5 mdp mdp NOUN bracis-19024 167 6 into into ADP bracis-19024 167 7 a a DET bracis-19024 167 8 semi semi ADJ bracis-19024 167 9 - - NOUN bracis-19024 167 10 mdp mdp ADJ bracis-19024 167 11     SPACE bracis-19024 167 12 [ [ X bracis-19024 167 13 14 14 NUM bracis-19024 167 14 ] ] PUNCT bracis-19024 167 15 . . PUNCT bracis-19024 168 1 to to PART bracis-19024 168 2 make make VERB bracis-19024 168 3 our our PRON bracis-19024 168 4 formulation formulation NOUN bracis-19024 168 5 compatible compatible ADJ bracis-19024 168 6 with with ADP bracis-19024 168 7 a a DET bracis-19024 168 8 semi semi ADJ bracis-19024 168 9 - - NOUN bracis-19024 168 10 mdp mdp NOUN bracis-19024 168 11 , , PUNCT bracis-19024 168 12 we we PRON bracis-19024 168 13 extend extend VERB bracis-19024 168 14 the the DET bracis-19024 168 15 reward reward NOUN bracis-19024 168 16 function function NOUN bracis-19024 168 17 to to PART bracis-19024 168 18 return return VERB bracis-19024 168 19 zero zero NUM bracis-19024 168 20 in in ADP bracis-19024 168 21 all all DET bracis-19024 168 22 intermediate intermediate ADJ bracis-19024 168 23 states state NOUN bracis-19024 168 24 after after ADP bracis-19024 168 25 successfully successfully ADV bracis-19024 168 26 scheduling schedule VERB bracis-19024 168 27 a a DET bracis-19024 168 28 job job NOUN bracis-19024 168 29 . . PUNCT bracis-19024 169 1 4.4 4.4 NUM bracis-19024 169 2 reducing reduce VERB bracis-19024 169 3 the the DET bracis-19024 169 4 noise noise NOUN bracis-19024 169 5 of of ADP bracis-19024 169 6 the the DET bracis-19024 169 7 reward reward NOUN bracis-19024 169 8 signal signal NOUN bracis-19024 169 9 based base VERB bracis-19024 169 10 on on ADP bracis-19024 169 11 the the DET bracis-19024 169 12 idea idea NOUN bracis-19024 169 13 of of ADP bracis-19024 169 14 only only ADV bracis-19024 169 15 showing show VERB bracis-19024 169 16 the the DET bracis-19024 169 17 agent agent NOUN bracis-19024 169 18 what what PRON bracis-19024 169 19 it it PRON bracis-19024 169 20 can can AUX bracis-19024 169 21 use use VERB bracis-19024 169 22 to to PART bracis-19024 169 23 learn learn VERB bracis-19024 169 24 and and CCONJ bracis-19024 169 25 act act VERB bracis-19024 169 26 , , PUNCT bracis-19024 169 27 we we PRON bracis-19024 169 28 noticed notice VERB bracis-19024 169 29 that that SCONJ bracis-19024 169 30 the the DET bracis-19024 169 31 reward reward NOUN bracis-19024 169 32 signal signal NOUN bracis-19024 169 33 could could AUX bracis-19024 169 34 be be AUX bracis-19024 169 35 further far ADV bracis-19024 169 36 improved improve VERB bracis-19024 169 37 by by ADP bracis-19024 169 38 , , PUNCT bracis-19024 169 39 instead instead ADV bracis-19024 169 40 of of ADP bracis-19024 169 41 computing compute VERB bracis-19024 169 42 the the DET bracis-19024 169 43 online online ADJ bracis-19024 169 44 slowdown slowdown NOUN bracis-19024 169 45 of of ADP bracis-19024 169 46 all all DET bracis-19024 169 47 jobs job NOUN bracis-19024 169 48 in in ADP bracis-19024 169 49 the the DET bracis-19024 169 50 system system NOUN bracis-19024 169 51 \(\mathcal \(\mathcal ADJ bracis-19024 169 52 { { PUNCT bracis-19024 169 53 j}\ j}\ NOUN bracis-19024 169 54 ) ) PUNCT bracis-19024 169 55 , , PUNCT bracis-19024 169 56 considering consider VERB bracis-19024 169 57 only only ADV bracis-19024 169 58 the the DET bracis-19024 169 59 jobs job NOUN bracis-19024 169 60 that that PRON bracis-19024 169 61 are be AUX bracis-19024 169 62 in in ADP bracis-19024 169 63 the the DET bracis-19024 169 64 waiting waiting NOUN bracis-19024 169 65 queue queue NOUN bracis-19024 169 66 , , PUNCT bracis-19024 169 67 and and CCONJ bracis-19024 169 68 within within ADP bracis-19024 169 69 the the DET bracis-19024 169 70 job job NOUN bracis-19024 169 71 slots slot NOUN bracis-19024 169 72 window window NOUN bracis-19024 169 73 w w PROPN bracis-19024 169 74 : : PUNCT bracis-19024 169 75 the the DET bracis-19024 169 76 jobs job NOUN bracis-19024 169 77 that that PRON bracis-19024 169 78 can can AUX bracis-19024 169 79 be be AUX bracis-19024 169 80 directly directly ADV bracis-19024 169 81 influenced influence VERB bracis-19024 169 82 by by ADP bracis-19024 169 83 the the DET bracis-19024 169 84 agent agent NOUN bracis-19024 169 85 ’s ’s PART bracis-19024 169 86 actions action NOUN bracis-19024 169 87 . . PUNCT bracis-19024 170 1 therefore therefore ADV bracis-19024 170 2 , , PUNCT bracis-19024 170 3 we we PRON bracis-19024 170 4 defined define VERB bracis-19024 170 5 the the DET bracis-19024 170 6 set set NOUN bracis-19024 170 7 \(\mathcal \(\mathcal ADJ bracis-19024 170 8 { { PUNCT bracis-19024 170 9 w}\ w}\ PROPN bracis-19024 170 10 ) ) PUNCT bracis-19024 170 11 that that PRON bracis-19024 170 12 contains contain VERB bracis-19024 170 13 the the DET bracis-19024 170 14 subset subset NOUN bracis-19024 170 15 of of ADP bracis-19024 170 16 jobs job NOUN bracis-19024 170 17 from from ADP bracis-19024 170 18 \(\mathcal \(\mathcal ADJ bracis-19024 170 19 { { PUNCT bracis-19024 170 20 j}\ j}\ NOUN bracis-19024 170 21 ) ) PUNCT bracis-19024 170 22 that that PRON bracis-19024 170 23 are be AUX bracis-19024 170 24 within within ADP bracis-19024 170 25 the the DET bracis-19024 170 26 window window NOUN bracis-19024 170 27 w w NOUN bracis-19024 170 28 , , PUNCT bracis-19024 170 29 and and CCONJ bracis-19024 170 30 the the DET bracis-19024 170 31 reward reward NOUN bracis-19024 170 32 function function NOUN bracis-19024 170 33 became become VERB bracis-19024 170 34 \(\mathcal \(\mathcal ADJ bracis-19024 170 35 { { PUNCT bracis-19024 170 36 r r NOUN bracis-19024 170 37 } } PUNCT bracis-19024 170 38 = = PUNCT bracis-19024 170 39 -\sum -\sum PROPN bracis-19024 170 40 _ _ PUNCT bracis-19024 170 41 { { PUNCT bracis-19024 170 42 j\in j\in PROPN bracis-19024 170 43 \mathcal \mathcal PROPN bracis-19024 170 44 { { PUNCT bracis-19024 170 45 w}}\frac{1}{t_e(j_i)}\ w}}\frac{1}{t_e(j_i)}\ PROPN bracis-19024 170 46 ) ) PUNCT bracis-19024 170 47 when when SCONJ bracis-19024 170 48 the the DET bracis-19024 170 49 action action NOUN bracis-19024 170 50 taken take VERB bracis-19024 170 51 does do AUX bracis-19024 170 52 n’t not PART bracis-19024 170 53 schedule schedule VERB bracis-19024 170 54 a a DET bracis-19024 170 55 job job NOUN bracis-19024 170 56 , , PUNCT bracis-19024 170 57 and and CCONJ bracis-19024 170 58 0 0 NUM bracis-19024 170 59 otherwise otherwise ADV bracis-19024 170 60 . . PUNCT bracis-19024 171 1 we we PRON bracis-19024 171 2 evaluate evaluate VERB bracis-19024 171 3 the the DET bracis-19024 171 4 impact impact NOUN bracis-19024 171 5 of of ADP bracis-19024 171 6 the the DET bracis-19024 171 7 various various ADJ bracis-19024 171 8 mdps mdps NOUN bracis-19024 171 9 on on ADP bracis-19024 171 10 agent agent NOUN bracis-19024 171 11 performance performance NOUN bracis-19024 171 12 by by ADP bracis-19024 171 13 performing perform VERB bracis-19024 171 14 two two NUM bracis-19024 171 15 sets set NOUN bracis-19024 171 16 of of ADP bracis-19024 171 17 experiments experiment NOUN bracis-19024 171 18 , , PUNCT bracis-19024 171 19 one one NUM bracis-19024 171 20 in in ADP bracis-19024 171 21 which which PRON bracis-19024 171 22 we we PRON bracis-19024 171 23 observe observe VERB bracis-19024 171 24 the the DET bracis-19024 171 25 impact impact NOUN bracis-19024 171 26 of of ADP bracis-19024 171 27 the the DET bracis-19024 171 28 changes change NOUN bracis-19024 171 29 proposed propose VERB bracis-19024 171 30 in in ADP bracis-19024 171 31 sects sect NOUN bracis-19024 171 32 . . PUNCT bracis-19024 171 33     SPACE bracis-19024 171 34 4.2 4.2 NUM bracis-19024 171 35 through through ADP bracis-19024 171 36     SPACE bracis-19024 171 37 4.4 4.4 NUM bracis-19024 171 38 ( ( PUNCT bracis-19024 171 39 called call VERB bracis-19024 171 40 compact compact ADJ bracis-19024 171 41 , , PUNCT bracis-19024 171 42 sparse sparse ADJ bracis-19024 171 43 , , PUNCT bracis-19024 171 44 and and CCONJ bracis-19024 171 45 reduced reduce VERB bracis-19024 171 46 respectively respectively ADV bracis-19024 171 47 in in ADP bracis-19024 171 48 the the DET bracis-19024 171 49 experiments experiment NOUN bracis-19024 171 50 ) ) PUNCT bracis-19024 171 51 as as SCONJ bracis-19024 171 52 opposed oppose VERB bracis-19024 171 53 to to ADP bracis-19024 171 54 the the DET bracis-19024 171 55 dense dense ADJ bracis-19024 171 56 mdp mdp NOUN bracis-19024 171 57 , , PUNCT bracis-19024 171 58 and and CCONJ bracis-19024 171 59 another another PRON bracis-19024 171 60 in in ADP bracis-19024 171 61 which which PRON bracis-19024 171 62 we we PRON bracis-19024 171 63 observe observe VERB bracis-19024 171 64 the the DET bracis-19024 171 65 impact impact NOUN bracis-19024 171 66 of of ADP bracis-19024 171 67 using use VERB bracis-19024 171 68 an an DET bracis-19024 171 69 event event NOUN bracis-19024 171 70 - - PUNCT bracis-19024 171 71 based base VERB bracis-19024 171 72 simulation simulation NOUN bracis-19024 171 73 and and CCONJ bracis-19024 171 74 bounded bound VERB bracis-19024 171 75 rewards reward NOUN bracis-19024 171 76 both both CCONJ bracis-19024 171 77 in in ADP bracis-19024 171 78 dense dense ADJ bracis-19024 171 79 and and CCONJ bracis-19024 171 80 compact compact ADJ bracis-19024 171 81 mdps mdps NOUN bracis-19024 171 82 . . PUNCT bracis-19024 172 1 5 5 NUM bracis-19024 172 2 experiments experiment NOUN bracis-19024 172 3 in in ADP bracis-19024 172 4 order order NOUN bracis-19024 172 5 to to PART bracis-19024 172 6 evaluate evaluate VERB bracis-19024 172 7 our our PRON bracis-19024 172 8 methodology methodology NOUN bracis-19024 172 9 , , PUNCT bracis-19024 172 10 we we PRON bracis-19024 172 11 used use VERB bracis-19024 172 12 open open ADJ bracis-19024 172 13 - - PUNCT bracis-19024 172 14 source source NOUN bracis-19024 172 15 libraries library NOUN bracis-19024 172 16 to to PART bracis-19024 172 17 implement implement VERB bracis-19024 172 18 both both PRON bracis-19024 172 19 our our PRON bracis-19024 172 20 agents agent NOUN bracis-19024 172 21 and and CCONJ bracis-19024 172 22 environment environment NOUN bracis-19024 172 23 , , PUNCT bracis-19024 172 24 with with ADP bracis-19024 172 25 stable stable ADJ bracis-19024 172 26 - - PUNCT bracis-19024 172 27 baselines3 baselines3 NOUN bracis-19024 172 28     SPACE bracis-19024 173 1 [ [ X bracis-19024 173 2 10 10 NUM bracis-19024 173 3 ] ] PUNCT bracis-19024 173 4 providing provide VERB bracis-19024 173 5 the the DET bracis-19024 173 6 ppo ppo PROPN bracis-19024 173 7 agent agent NOUN bracis-19024 173 8 and and CCONJ bracis-19024 173 9 its its PRON bracis-19024 173 10 training training NOUN bracis-19024 173 11 loop loop NOUN bracis-19024 173 12 , , PUNCT bracis-19024 173 13 and and CCONJ bracis-19024 173 14 sched sche VERB bracis-19024 173 15 - - PUNCT bracis-19024 173 16 rl rl NOUN bracis-19024 173 17 - - PUNCT bracis-19024 173 18 gym gym NOUN bracis-19024 173 19     SPACE bracis-19024 174 1 [ [ X bracis-19024 174 2 2 2 X bracis-19024 174 3 ] ] PUNCT bracis-19024 174 4 providing provide VERB bracis-19024 174 5 the the DET bracis-19024 174 6 simulator simulator NOUN bracis-19024 174 7 and and CCONJ bracis-19024 174 8 environment environment NOUN bracis-19024 174 9 implementation implementation NOUN bracis-19024 174 10 . . PUNCT bracis-19024 175 1 table table NOUN bracis-19024 175 2 2 2 NUM bracis-19024 175 3 . . X bracis-19024 175 4 list list NOUN bracis-19024 175 5 of of ADP bracis-19024 175 6 hyper hyper NOUN bracis-19024 175 7 - - NOUN bracis-19024 175 8 parameters parameter NOUN bracis-19024 175 9 used use VERB bracis-19024 175 10 when when SCONJ bracis-19024 175 11 training train VERB bracis-19024 175 12 agents.full agents.full ADJ bracis-19024 175 13 size size NOUN bracis-19024 175 14 table table NOUN bracis-19024 175 15 all all DET bracis-19024 175 16 our our PRON bracis-19024 175 17 experiments experiment NOUN bracis-19024 175 18 consisted consist VERB bracis-19024 175 19 of of ADP bracis-19024 175 20 training train VERB bracis-19024 175 21 a a DET bracis-19024 175 22 ppo ppo PROPN bracis-19024 175 23 agent agent NOUN bracis-19024 175 24 in in ADP bracis-19024 175 25 the the DET bracis-19024 175 26 different different ADJ bracis-19024 175 27 formulations formulation NOUN bracis-19024 175 28 of of ADP bracis-19024 175 29 the the DET bracis-19024 175 30 previous previous ADJ bracis-19024 175 31 section section NOUN bracis-19024 175 32 . . PUNCT bracis-19024 176 1 we we PRON bracis-19024 176 2 also also ADV bracis-19024 176 3 fixed fix VERB bracis-19024 176 4 the the DET bracis-19024 176 5 neural neural ADJ bracis-19024 176 6 network network NOUN bracis-19024 176 7 architecture architecture NOUN bracis-19024 176 8 used use VERB bracis-19024 176 9 for for ADP bracis-19024 176 10 function function NOUN bracis-19024 176 11 approximation approximation NOUN bracis-19024 176 12 , , PUNCT bracis-19024 176 13 consisting consist VERB bracis-19024 176 14 of of ADP bracis-19024 176 15 a a DET bracis-19024 176 16 two two NUM bracis-19024 176 17 - - PUNCT bracis-19024 176 18 layer layer NOUN bracis-19024 176 19 neural neural ADJ bracis-19024 176 20 network network NOUN bracis-19024 176 21 with with ADP bracis-19024 176 22 64 64 NUM bracis-19024 176 23 units unit NOUN bracis-19024 176 24 in in ADP bracis-19024 176 25 each each DET bracis-19024 176 26 layer layer NOUN bracis-19024 176 27 , , PUNCT bracis-19024 176 28 and and CCONJ bracis-19024 176 29 with with ADP bracis-19024 176 30 parameter parameter NOUN bracis-19024 176 31 sharing sharing NOUN bracis-19024 176 32 between between ADP bracis-19024 176 33 policy policy NOUN bracis-19024 176 34 and and CCONJ bracis-19024 176 35 value value NOUN bracis-19024 176 36 networks network NOUN bracis-19024 176 37 . . PUNCT bracis-19024 177 1 the the DET bracis-19024 177 2 fixed fix VERB bracis-19024 177 3 number number NOUN bracis-19024 177 4 of of ADP bracis-19024 177 5 units unit NOUN bracis-19024 177 6 implies imply VERB bracis-19024 177 7 the the DET bracis-19024 177 8 image image NOUN bracis-19024 177 9 - - PUNCT bracis-19024 177 10 like like ADJ bracis-19024 177 11 representation representation NOUN bracis-19024 177 12 will will AUX bracis-19024 177 13 use use VERB bracis-19024 177 14 more more ADJ bracis-19024 177 15 parameters parameter NOUN bracis-19024 177 16 , , PUNCT bracis-19024 177 17 as as SCONJ bracis-19024 177 18 it it PRON bracis-19024 177 19 contains contain VERB bracis-19024 177 20 more more ADJ bracis-19024 177 21 data datum NOUN bracis-19024 177 22 than than ADP bracis-19024 177 23 the the DET bracis-19024 177 24 compact compact ADJ bracis-19024 177 25 representation representation NOUN bracis-19024 177 26 . . PUNCT bracis-19024 178 1 the the DET bracis-19024 178 2 hyper hyper NOUN bracis-19024 178 3 - - NOUN bracis-19024 178 4 parameters parameter NOUN bracis-19024 178 5 used use VERB bracis-19024 178 6 for for ADP bracis-19024 178 7 training train VERB bracis-19024 178 8 the the DET bracis-19024 178 9 agent agent NOUN bracis-19024 178 10 are be AUX bracis-19024 178 11 summarized summarize VERB bracis-19024 178 12 in in ADP bracis-19024 178 13 table table NOUN bracis-19024 178 14     SPACE bracis-19024 178 15 2 2 NUM bracis-19024 178 16 . . PUNCT bracis-19024 179 1 we we PRON bracis-19024 179 2 performed perform VERB bracis-19024 179 3 no no DET bracis-19024 179 4 hyper hyper ADJ bracis-19024 179 5 - - ADJ bracis-19024 179 6 parameter parameter ADJ bracis-19024 179 7 optimization optimization NOUN bracis-19024 179 8 , , PUNCT bracis-19024 179 9 and and CCONJ bracis-19024 179 10 used use VERB bracis-19024 179 11 values value NOUN bracis-19024 179 12 found find VERB bracis-19024 179 13 in in ADP bracis-19024 179 14 the the DET bracis-19024 179 15 literature literature NOUN bracis-19024 179 16 when when SCONJ bracis-19024 179 17 training train VERB bracis-19024 179 18 the the DET bracis-19024 179 19 image image NOUN bracis-19024 179 20 - - PUNCT bracis-19024 179 21 like like ADJ bracis-19024 179 22 agent agent NOUN bracis-19024 179 23 . . PUNCT bracis-19024 180 1 for for ADP bracis-19024 180 2 a a DET bracis-19024 180 3 full full ADJ bracis-19024 180 4 description description NOUN bracis-19024 180 5 of of ADP bracis-19024 180 6 ppo ppo PROPN bracis-19024 180 7 , , PUNCT bracis-19024 180 8 we we PRON bracis-19024 180 9 direct direct VERB bracis-19024 180 10 the the DET bracis-19024 180 11 reader reader NOUN bracis-19024 180 12 to to ADP bracis-19024 180 13 schulman schulman PROPN bracis-19024 180 14 et et PROPN bracis-19024 180 15 al al PROPN bracis-19024 180 16 . . PUNCT bracis-19024 180 17     SPACE bracis-19024 181 1 [ [ X bracis-19024 181 2 12 12 NUM bracis-19024 181 3 ] ] PUNCT bracis-19024 181 4 . . PUNCT bracis-19024 182 1 we we PRON bracis-19024 182 2 also also ADV bracis-19024 182 3 maintained maintain VERB bracis-19024 182 4 the the DET bracis-19024 182 5 environment environment NOUN bracis-19024 182 6 specification specification NOUN bracis-19024 182 7 fixed fix VERB bracis-19024 182 8 for for ADP bracis-19024 182 9 all all DET bracis-19024 182 10 agent agent NOUN bracis-19024 182 11 evaluations evaluation NOUN bracis-19024 182 12 and and CCONJ bracis-19024 182 13 used use VERB bracis-19024 182 14 \(w=10\ \(w=10\ PROPN bracis-19024 182 15 ) ) PUNCT bracis-19024 182 16 job job NOUN bracis-19024 182 17 slots slot NOUN bracis-19024 182 18 , , PUNCT bracis-19024 182 19 with with ADP bracis-19024 182 20 simulations simulation NOUN bracis-19024 182 21 of of ADP bracis-19024 182 22 length length NOUN bracis-19024 182 23 \(t=100\ \(t=100\ PROPN bracis-19024 182 24 ) ) PUNCT bracis-19024 182 25 time time NOUN bracis-19024 182 26 - - PUNCT bracis-19024 182 27 steps step NOUN bracis-19024 182 28 and and CCONJ bracis-19024 182 29 time time NOUN bracis-19024 182 30 horizon horizon PROPN bracis-19024 182 31 \(h\in \(h\in PUNCT bracis-19024 182 32 \{20 \{20 PROPN bracis-19024 182 33 , , PUNCT bracis-19024 182 34 60\}\ 60\}\ NUM bracis-19024 182 35 ) ) PUNCT bracis-19024 182 36 . . PUNCT bracis-19024 183 1 these these DET bracis-19024 183 2 two two NUM bracis-19024 183 3 horizon horizon NOUN bracis-19024 183 4 values value NOUN bracis-19024 183 5 enable enable VERB bracis-19024 183 6 us we PRON bracis-19024 183 7 to to PART bracis-19024 183 8 contrast contrast VERB bracis-19024 183 9 cases case NOUN bracis-19024 183 10 in in ADP bracis-19024 183 11 which which PRON bracis-19024 183 12 agents agent NOUN bracis-19024 183 13 can can AUX bracis-19024 183 14 see see VERB bracis-19024 183 15 when when SCONJ bracis-19024 183 16 jobs job NOUN bracis-19024 183 17 will will AUX bracis-19024 183 18 complete complete VERB bracis-19024 183 19 , , PUNCT bracis-19024 183 20 or or CCONJ bracis-19024 183 21 not not PART bracis-19024 183 22 . . PUNCT bracis-19024 184 1 regarding regard VERB bracis-19024 184 2 the the DET bracis-19024 184 3 workload workload NOUN bracis-19024 184 4 , , PUNCT bracis-19024 184 5 we we PRON bracis-19024 184 6 used use VERB bracis-19024 184 7 a a DET bracis-19024 184 8 workload workload NOUN bracis-19024 184 9 generator generator NOUN bracis-19024 184 10 from from ADP bracis-19024 184 11 the the DET bracis-19024 184 12 literature literature NOUN bracis-19024 184 13     SPACE bracis-19024 185 1 [ [ X bracis-19024 185 2 2 2 NUM bracis-19024 185 3 , , PUNCT bracis-19024 185 4 7 7 NUM bracis-19024 185 5 ] ] PUNCT bracis-19024 185 6 , , PUNCT bracis-19024 185 7 which which PRON bracis-19024 185 8 submitted submit VERB bracis-19024 185 9 a a DET bracis-19024 185 10 new new ADJ bracis-19024 185 11 job job NOUN bracis-19024 185 12 with with ADP bracis-19024 185 13 \(30\%\ \(30\%\ NOUN bracis-19024 185 14 ) ) PUNCT bracis-19024 185 15 chance chance NOUN bracis-19024 185 16 on on ADP bracis-19024 185 17 each each DET bracis-19024 185 18 time time NOUN bracis-19024 185 19 step step NOUN bracis-19024 185 20 . . PUNCT bracis-19024 186 1 of of ADP bracis-19024 186 2 these these PRON bracis-19024 186 3 , , PUNCT bracis-19024 186 4 a a DET bracis-19024 186 5 job job NOUN bracis-19024 186 6 had have VERB bracis-19024 186 7 \(80\%\ \(80\%\ NOUN bracis-19024 186 8 ) ) PUNCT bracis-19024 186 9 chance chance NOUN bracis-19024 186 10 of of ADP bracis-19024 186 11 being be AUX bracis-19024 186 12 a a DET bracis-19024 186 13 “ " PUNCT bracis-19024 186 14 small small ADJ bracis-19024 186 15 ” " PUNCT bracis-19024 186 16 job job NOUN bracis-19024 186 17 , , PUNCT bracis-19024 186 18 and and CCONJ bracis-19024 186 19 “ " PUNCT bracis-19024 186 20 large large ADJ bracis-19024 186 21 ” " PUNCT bracis-19024 186 22 otherwise otherwise ADV bracis-19024 186 23 . . PUNCT bracis-19024 187 1 the the DET bracis-19024 187 2 number number NOUN bracis-19024 187 3 of of ADP bracis-19024 187 4 processors processor NOUN bracis-19024 187 5 \(n_p\ \(n_p\ PROPN bracis-19024 187 6 ) ) PUNCT bracis-19024 187 7 was be AUX bracis-19024 187 8 chosen choose VERB bracis-19024 187 9 in in ADP bracis-19024 187 10 the the DET bracis-19024 187 11 set set NOUN bracis-19024 187 12 { { PUNCT bracis-19024 187 13 10 10 NUM bracis-19024 187 14 , , PUNCT bracis-19024 187 15 32 32 NUM bracis-19024 187 16 , , PUNCT bracis-19024 187 17 64 64 NUM bracis-19024 187 18 } } PUNCT bracis-19024 187 19 , , PUNCT bracis-19024 187 20 while while SCONJ bracis-19024 187 21 the the DET bracis-19024 187 22 maximum maximum ADJ bracis-19024 187 23 job job NOUN bracis-19024 187 24 length length NOUN bracis-19024 187 25 ( ( PUNCT bracis-19024 187 26 duration duration NOUN bracis-19024 187 27 ) ) PUNCT bracis-19024 187 28 d d NOUN bracis-19024 187 29 varied varied ADJ bracis-19024 187 30 from from ADP bracis-19024 187 31 { { PUNCT bracis-19024 187 32 15 15 NUM bracis-19024 187 33 , , PUNCT bracis-19024 187 34 33 33 NUM bracis-19024 187 35 , , PUNCT bracis-19024 187 36 48 48 NUM bracis-19024 187 37 } } PUNCT bracis-19024 187 38 and and CCONJ bracis-19024 187 39 the the DET bracis-19024 187 40 size size NOUN bracis-19024 187 41 of of ADP bracis-19024 187 42 the the DET bracis-19024 187 43 largest large ADJ bracis-19024 187 44 job job NOUN bracis-19024 187 45 ( ( PUNCT bracis-19024 187 46 number number NOUN bracis-19024 187 47 of of ADP bracis-19024 187 48 processors processor NOUN bracis-19024 187 49 ) ) PUNCT bracis-19024 187 50 \(j_s\ \(j_s\ ADJ bracis-19024 187 51 ) ) PUNCT bracis-19024 187 52 came come VERB bracis-19024 187 53 from from ADP bracis-19024 187 54 the the DET bracis-19024 187 55 set set NOUN bracis-19024 187 56 { { PUNCT bracis-19024 187 57 10 10 NUM bracis-19024 187 58 , , PUNCT bracis-19024 187 59 32 32 NUM bracis-19024 187 60 , , PUNCT bracis-19024 187 61 64 64 NUM bracis-19024 187 62 } } PUNCT bracis-19024 187 63 . . PUNCT bracis-19024 188 1 in in ADP bracis-19024 188 2 the the DET bracis-19024 188 3 workload workload NOUN bracis-19024 188 4 generator generator NOUN bracis-19024 188 5 , , PUNCT bracis-19024 188 6 the the DET bracis-19024 188 7 length length NOUN bracis-19024 188 8 of of ADP bracis-19024 188 9 small small ADJ bracis-19024 188 10 jobs job NOUN bracis-19024 188 11 was be AUX bracis-19024 188 12 sampled sample VERB bracis-19024 188 13 uniformly uniformly ADV bracis-19024 188 14 from from ADP bracis-19024 188 15 , , PUNCT bracis-19024 188 16 and and CCONJ bracis-19024 188 17 the the DET bracis-19024 188 18 length length NOUN bracis-19024 188 19 of of ADP bracis-19024 188 20 large large ADJ bracis-19024 188 21 jobs job NOUN bracis-19024 188 22 was be AUX bracis-19024 188 23 sampled sample VERB bracis-19024 188 24 uniformly uniformly ADV bracis-19024 188 25 from from ADP bracis-19024 188 26 . . PUNCT bracis-19024 189 1 the the DET bracis-19024 189 2 number number NOUN bracis-19024 189 3 of of ADP bracis-19024 189 4 processors processor NOUN bracis-19024 189 5 used use VERB bracis-19024 189 6 by by ADP bracis-19024 189 7 any any DET bracis-19024 189 8 job job NOUN bracis-19024 189 9 was be AUX bracis-19024 189 10 sampled sample VERB bracis-19024 189 11 from from ADP bracis-19024 189 12 . . PUNCT bracis-19024 190 1 all all DET bracis-19024 190 2 agents agent NOUN bracis-19024 190 3 were be AUX bracis-19024 190 4 trained train VERB bracis-19024 190 5 for for ADP bracis-19024 190 6 three three NUM bracis-19024 190 7 million million NUM bracis-19024 190 8 time time NOUN bracis-19024 190 9 - - PUNCT bracis-19024 190 10 steps step NOUN bracis-19024 190 11 as as SCONJ bracis-19024 190 12 perceived perceive VERB bracis-19024 190 13 by by ADP bracis-19024 190 14 the the DET bracis-19024 190 15 agent agent NOUN bracis-19024 190 16 . . PUNCT bracis-19024 191 1 this this PRON bracis-19024 191 2 means mean VERB bracis-19024 191 3 that that SCONJ bracis-19024 191 4 all all DET bracis-19024 191 5 agents agent NOUN bracis-19024 191 6 will will AUX bracis-19024 191 7 see see VERB bracis-19024 191 8 the the DET bracis-19024 191 9 same same ADJ bracis-19024 191 10 number number NOUN bracis-19024 191 11 of of ADP bracis-19024 191 12 states state NOUN bracis-19024 191 13 , , PUNCT bracis-19024 191 14 and and CCONJ bracis-19024 191 15 will will AUX bracis-19024 191 16 take take VERB bracis-19024 191 17 the the DET bracis-19024 191 18 same same ADJ bracis-19024 191 19 number number NOUN bracis-19024 191 20 of of ADP bracis-19024 191 21 actions action NOUN bracis-19024 191 22 , , PUNCT bracis-19024 191 23 but but CCONJ bracis-19024 191 24 the the DET bracis-19024 191 25 number number NOUN bracis-19024 191 26 of of ADP bracis-19024 191 27 time time NOUN bracis-19024 191 28 steps step NOUN bracis-19024 191 29 in in ADP bracis-19024 191 30 the the DET bracis-19024 191 31 underlying underlying ADJ bracis-19024 191 32 simulation simulation NOUN bracis-19024 191 33 will will AUX bracis-19024 191 34 vary vary VERB bracis-19024 191 35 , , PUNCT bracis-19024 191 36 due due ADP bracis-19024 191 37 to to ADP bracis-19024 191 38 the the DET bracis-19024 191 39 event event NOUN bracis-19024 191 40 - - PUNCT bracis-19024 191 41 based base VERB bracis-19024 191 42 case case NOUN bracis-19024 191 43 becoming become VERB bracis-19024 191 44 a a DET bracis-19024 191 45 semi semi NOUN bracis-19024 191 46 - - NOUN bracis-19024 191 47 mdp mdp NOUN bracis-19024 191 48 . . PUNCT bracis-19024 192 1 we we PRON bracis-19024 192 2 evaluated evaluate VERB bracis-19024 192 3 agents agent NOUN bracis-19024 192 4 with with ADP bracis-19024 192 5 a a DET bracis-19024 192 6 thousand thousand NUM bracis-19024 192 7 independent independent ADJ bracis-19024 192 8 trials trial NOUN bracis-19024 192 9 , , PUNCT bracis-19024 192 10 reporting report VERB bracis-19024 192 11 average average ADJ bracis-19024 192 12 values value NOUN bracis-19024 192 13 . . PUNCT bracis-19024 193 1 fig fig NOUN bracis-19024 193 2 . . PUNCT bracis-19024 194 1 3 3 X bracis-19024 194 2 . . X bracis-19024 194 3 learning learn VERB bracis-19024 194 4 curves curve NOUN bracis-19024 194 5 for for ADP bracis-19024 194 6 various various ADJ bracis-19024 194 7 scenarios scenario NOUN bracis-19024 194 8 with with ADP bracis-19024 194 9 \(h=20\ \(h=20\ NOUN bracis-19024 194 10 ) ) PUNCT bracis-19024 194 11 contrasting contrast VERB bracis-19024 194 12 learning learning NOUN bracis-19024 194 13 using use VERB bracis-19024 194 14 a a DET bracis-19024 194 15 compact compact ADJ bracis-19024 194 16 representation representation NOUN bracis-19024 194 17 with with ADP bracis-19024 194 18 learning learn VERB bracis-19024 194 19 with with ADP bracis-19024 194 20 an an DET bracis-19024 194 21 image image NOUN bracis-19024 194 22 - - PUNCT bracis-19024 194 23 like like ADJ bracis-19024 194 24 representation representation NOUN bracis-19024 194 25 . . PUNCT bracis-19024 195 1 curves curve NOUN bracis-19024 195 2 are be AUX bracis-19024 195 3 an an DET bracis-19024 195 4 average average NOUN bracis-19024 195 5 of of ADP bracis-19024 195 6 six six NUM bracis-19024 195 7 agents agent NOUN bracis-19024 195 8 , , PUNCT bracis-19024 195 9 with with ADP bracis-19024 195 10 shaded shaded ADJ bracis-19024 195 11 areas area NOUN bracis-19024 195 12 representing represent VERB bracis-19024 195 13 one one NUM bracis-19024 195 14 standard standard ADJ bracis-19024 195 15 deviation deviation NOUN bracis-19024 195 16 , , PUNCT bracis-19024 195 17 and and CCONJ bracis-19024 195 18 show show VERB bracis-19024 195 19 a a DET bracis-19024 195 20 moving move VERB bracis-19024 195 21 average average NOUN bracis-19024 195 22 of of ADP bracis-19024 195 23 total total ADJ bracis-19024 195 24 rewards reward NOUN bracis-19024 195 25 received receive VERB bracis-19024 195 26 by by ADP bracis-19024 195 27 the the DET bracis-19024 195 28 agents agent NOUN bracis-19024 195 29 during during ADP bracis-19024 195 30 training training NOUN bracis-19024 195 31 . . PUNCT bracis-19024 196 1 full full ADJ bracis-19024 196 2 size size NOUN bracis-19024 196 3 image image NOUN bracis-19024 196 4 in in ADP bracis-19024 196 5 fig fig NOUN bracis-19024 196 6 . . PUNCT bracis-19024 196 7     SPACE bracis-19024 196 8 3 3 NUM bracis-19024 196 9 we we PRON bracis-19024 196 10 show show VERB bracis-19024 196 11 a a DET bracis-19024 196 12 sampling sampling NOUN bracis-19024 196 13 of of ADP bracis-19024 196 14 learning learn VERB bracis-19024 196 15 curves curve NOUN bracis-19024 196 16 comparing compare VERB bracis-19024 196 17 the the DET bracis-19024 196 18 learning learn VERB bracis-19024 196 19 performance performance NOUN bracis-19024 196 20 of of ADP bracis-19024 196 21 agents agent NOUN bracis-19024 196 22 that that PRON bracis-19024 196 23 were be AUX bracis-19024 196 24 trained train VERB bracis-19024 196 25 using use VERB bracis-19024 196 26 the the DET bracis-19024 196 27 image image NOUN bracis-19024 196 28 - - PUNCT bracis-19024 196 29 like like ADJ bracis-19024 196 30 representation representation NOUN bracis-19024 196 31 and and CCONJ bracis-19024 196 32 the the DET bracis-19024 196 33 compact compact ADJ bracis-19024 196 34 representation representation NOUN bracis-19024 196 35 with with ADP bracis-19024 196 36 rewards reward NOUN bracis-19024 196 37 computed compute VERB bracis-19024 196 38 from from ADP bracis-19024 196 39 all all DET bracis-19024 196 40 jobs job NOUN bracis-19024 196 41 . . PUNCT bracis-19024 197 1 the the DET bracis-19024 197 2 compact compact ADJ bracis-19024 197 3 representation representation NOUN bracis-19024 197 4 converges converge VERB bracis-19024 197 5 faster fast ADV bracis-19024 197 6 than than ADP bracis-19024 197 7 the the DET bracis-19024 197 8 image image NOUN bracis-19024 197 9 - - PUNCT bracis-19024 197 10 like like ADJ bracis-19024 197 11 representation representation NOUN bracis-19024 197 12 , , PUNCT bracis-19024 197 13 probably probably ADV bracis-19024 197 14 due due ADP bracis-19024 197 15 to to ADP bracis-19024 197 16 its its PRON bracis-19024 197 17 smaller small ADJ bracis-19024 197 18 number number NOUN bracis-19024 197 19 of of ADP bracis-19024 197 20 parameters parameter NOUN bracis-19024 197 21 . . PUNCT bracis-19024 198 1 we we PRON bracis-19024 198 2 also also ADV bracis-19024 198 3 notice notice VERB bracis-19024 198 4 that that SCONJ bracis-19024 198 5 although although SCONJ bracis-19024 198 6 convergence convergence NOUN bracis-19024 198 7 is be AUX bracis-19024 198 8 faster fast ADJ bracis-19024 198 9 , , PUNCT bracis-19024 198 10 the the DET bracis-19024 198 11 compact compact ADJ bracis-19024 198 12 representation representation NOUN bracis-19024 198 13 is be AUX bracis-19024 198 14 not not PART bracis-19024 198 15 necessarily necessarily ADV bracis-19024 198 16 better well ADJ bracis-19024 198 17 ( ( PUNCT bracis-19024 198 18 fig fig NOUN bracis-19024 198 19 . . PUNCT bracis-19024 198 20     SPACE bracis-19024 198 21 3c 3c NUM bracis-19024 198 22 , , PUNCT bracis-19024 198 23     SPACE bracis-19024 198 24 3d 3d NUM bracis-19024 198 25 ) ) PUNCT bracis-19024 198 26 . . PUNCT bracis-19024 199 1 there there PRON bracis-19024 199 2 does do AUX bracis-19024 199 3 n’t not PART bracis-19024 199 4 seem seem VERB bracis-19024 199 5 to to PART bracis-19024 199 6 be be AUX bracis-19024 199 7 a a DET bracis-19024 199 8 general general ADJ bracis-19024 199 9 rule rule NOUN bracis-19024 199 10 , , PUNCT bracis-19024 199 11 but but CCONJ bracis-19024 199 12 we we PRON bracis-19024 199 13 noticed notice VERB bracis-19024 199 14 that that SCONJ bracis-19024 199 15 when when SCONJ bracis-19024 199 16 jobs job NOUN bracis-19024 199 17 are be AUX bracis-19024 199 18 shorter short ADJ bracis-19024 199 19 ( ( PUNCT bracis-19024 199 20 the the DET bracis-19024 199 21 d d PROPN bracis-19024 199 22 parameter parameter NOUN bracis-19024 199 23 is be AUX bracis-19024 199 24 smaller small ADJ bracis-19024 199 25 ) ) PUNCT bracis-19024 199 26 , , PUNCT bracis-19024 199 27 the the DET bracis-19024 199 28 compact compact ADJ bracis-19024 199 29 representation representation NOUN bracis-19024 199 30 dominates dominate VERB bracis-19024 199 31 ( ( PUNCT bracis-19024 199 32 fig fig NOUN bracis-19024 199 33 . . PUNCT bracis-19024 199 34     SPACE bracis-19024 199 35 3a 3a NUM bracis-19024 199 36 , , PUNCT bracis-19024 199 37     SPACE bracis-19024 199 38 3b 3b NUM bracis-19024 199 39 ) ) PUNCT bracis-19024 199 40 . . PUNCT bracis-19024 200 1 when when SCONJ bracis-19024 200 2 d d NOUN bracis-19024 200 3 increases increase NOUN bracis-19024 200 4 and and CCONJ bracis-19024 200 5 most most ADJ bracis-19024 200 6 jobs job NOUN bracis-19024 200 7 use use VERB bracis-19024 200 8 few few ADJ bracis-19024 200 9 processors processor NOUN bracis-19024 200 10 ( ( PUNCT bracis-19024 200 11 \(j_s \(j_s NOUN bracis-19024 200 12 \ll \ll ADJ bracis-19024 200 13 n_p\ n_p\ NOUN bracis-19024 200 14 ) ) PUNCT bracis-19024 200 15 ) ) PUNCT bracis-19024 200 16 , , PUNCT bracis-19024 200 17 the the DET bracis-19024 200 18 compact compact ADJ bracis-19024 200 19 representation representation NOUN bracis-19024 200 20 tends tend VERB bracis-19024 200 21 to to PART bracis-19024 200 22 have have VERB bracis-19024 200 23 comparable comparable ADJ bracis-19024 200 24 performance performance NOUN bracis-19024 200 25 with with ADP bracis-19024 200 26 the the DET bracis-19024 200 27 image image NOUN bracis-19024 200 28 - - PUNCT bracis-19024 200 29 like like ADJ bracis-19024 200 30 representation representation NOUN bracis-19024 200 31 ( ( PUNCT bracis-19024 200 32 fig fig NOUN bracis-19024 200 33 . . PUNCT bracis-19024 200 34     SPACE bracis-19024 200 35 3c 3c NUM bracis-19024 200 36 ) ) PUNCT bracis-19024 200 37 , , PUNCT bracis-19024 200 38 whereas whereas SCONJ bracis-19024 200 39 when when SCONJ bracis-19024 200 40 jobs job NOUN bracis-19024 200 41 use use VERB bracis-19024 200 42 many many ADJ bracis-19024 200 43 processors processor NOUN bracis-19024 200 44 and and CCONJ bracis-19024 200 45 have have VERB bracis-19024 200 46 a a DET bracis-19024 200 47 longer long ADJ bracis-19024 200 48 duration duration NOUN bracis-19024 200 49 , , PUNCT bracis-19024 200 50 agents agent NOUN bracis-19024 200 51 using use VERB bracis-19024 200 52 the the DET bracis-19024 200 53 image image NOUN bracis-19024 200 54 - - PUNCT bracis-19024 200 55 like like ADJ bracis-19024 200 56 representation representation NOUN bracis-19024 200 57 learn learn VERB bracis-19024 200 58 the the DET bracis-19024 200 59 environment environment NOUN bracis-19024 200 60 better well ADV bracis-19024 200 61 ( ( PUNCT bracis-19024 200 62 fig fig NOUN bracis-19024 200 63 . . PUNCT bracis-19024 200 64     SPACE bracis-19024 200 65 3d 3d NUM bracis-19024 200 66 ) ) PUNCT bracis-19024 200 67 . . PUNCT bracis-19024 201 1 for for ADP bracis-19024 201 2 this this DET bracis-19024 201 3 set set NOUN bracis-19024 201 4 of of ADP bracis-19024 201 5 experiments experiment NOUN bracis-19024 201 6 , , PUNCT bracis-19024 201 7 the the DET bracis-19024 201 8 size size NOUN bracis-19024 201 9 of of ADP bracis-19024 201 10 the the DET bracis-19024 201 11 time time NOUN bracis-19024 201 12 horizon horizon NOUN bracis-19024 201 13 ( ( PUNCT bracis-19024 201 14 h h NOUN bracis-19024 201 15 ) ) PUNCT bracis-19024 201 16 does do AUX bracis-19024 201 17 n’t not PART bracis-19024 201 18 impact impact VERB bracis-19024 201 19 the the DET bracis-19024 201 20 learning learning NOUN bracis-19024 201 21 performance performance NOUN bracis-19024 201 22 , , PUNCT bracis-19024 201 23 as as SCONJ bracis-19024 201 24 curves curve NOUN bracis-19024 201 25 obtained obtain VERB bracis-19024 201 26 with with ADP bracis-19024 201 27 \(h=60\ \(h=60\ NOUN bracis-19024 201 28 ) ) PUNCT bracis-19024 201 29 ( ( PUNCT bracis-19024 201 30 not not PART bracis-19024 201 31 shown show VERB bracis-19024 201 32 ) ) PUNCT bracis-19024 201 33 are be AUX bracis-19024 201 34 indistinguishable indistinguishable ADJ bracis-19024 201 35 from from ADP bracis-19024 201 36 visual visual ADJ bracis-19024 201 37 inspection inspection NOUN bracis-19024 201 38 to to ADP bracis-19024 201 39 the the DET bracis-19024 201 40 ones one NOUN bracis-19024 201 41 obtained obtain VERB bracis-19024 201 42 with with ADP bracis-19024 201 43 \(h=20\ \(h=20\ NOUN bracis-19024 201 44 ) ) PUNCT bracis-19024 201 45 . . PUNCT bracis-19024 202 1 when when SCONJ bracis-19024 202 2 evaluating evaluate VERB bracis-19024 202 3 agents agent NOUN bracis-19024 202 4 , , PUNCT bracis-19024 202 5 we we PRON bracis-19024 202 6 performed perform VERB bracis-19024 202 7 t t NOUN bracis-19024 202 8 - - PUNCT bracis-19024 202 9 tests test NOUN bracis-19024 202 10 to to PART bracis-19024 202 11 check check VERB bracis-19024 202 12 whether whether SCONJ bracis-19024 202 13 there there PRON bracis-19024 202 14 was be VERB bracis-19024 202 15 a a DET bracis-19024 202 16 difference difference NOUN bracis-19024 202 17 in in ADP bracis-19024 202 18 agent agent NOUN bracis-19024 202 19 performance performance NOUN bracis-19024 202 20 when when SCONJ bracis-19024 202 21 using use VERB bracis-19024 202 22 these these DET bracis-19024 202 23 different different ADJ bracis-19024 202 24 h h NOUN bracis-19024 202 25 values value NOUN bracis-19024 202 26 . . PUNCT bracis-19024 203 1 in in ADP bracis-19024 203 2 other other ADJ bracis-19024 203 3 words word NOUN bracis-19024 203 4 , , PUNCT bracis-19024 203 5 the the DET bracis-19024 203 6 null null ADJ bracis-19024 203 7 hypothesis hypothesis NOUN bracis-19024 203 8 was be AUX bracis-19024 203 9 that that DET bracis-19024 203 10 performance performance NOUN bracis-19024 203 11 was be AUX bracis-19024 203 12 equal equal ADJ bracis-19024 203 13 , , PUNCT bracis-19024 203 14 and and CCONJ bracis-19024 203 15 the the DET bracis-19024 203 16 alternative alternative ADJ bracis-19024 203 17 hypothesis hypothesis NOUN bracis-19024 203 18 was be AUX bracis-19024 203 19 that that PRON bracis-19024 203 20 agent agent NOUN bracis-19024 203 21 performance performance NOUN bracis-19024 203 22 varied varied ADJ bracis-19024 203 23 . . PUNCT bracis-19024 204 1 in in ADP bracis-19024 204 2 this this DET bracis-19024 204 3 setting setting NOUN bracis-19024 204 4 , , PUNCT bracis-19024 204 5 the the DET bracis-19024 204 6 null null ADJ bracis-19024 204 7 hypothesis hypothesis NOUN bracis-19024 204 8 was be AUX bracis-19024 204 9 rejected reject VERB bracis-19024 204 10 only only ADV bracis-19024 204 11 \(36.6\%\ \(36.6\%\ NOUN bracis-19024 204 12 ) ) PUNCT bracis-19024 204 13 of of ADP bracis-19024 204 14 the the DET bracis-19024 204 15 time time NOUN bracis-19024 204 16 when when SCONJ bracis-19024 204 17 considering consider VERB bracis-19024 204 18 p p NOUN bracis-19024 204 19 - - PUNCT bracis-19024 204 20 values value NOUN bracis-19024 204 21     SPACE bracis-19024 204 22 \(\le \(\le PROPN bracis-19024 204 23 1\%\ 1\%\ NUM bracis-19024 204 24 ) ) PUNCT bracis-19024 204 25 . . PUNCT bracis-19024 205 1 fig fig NOUN bracis-19024 205 2 . . PUNCT bracis-19024 206 1 4 4 X bracis-19024 206 2 . . X bracis-19024 206 3 average average ADJ bracis-19024 206 4 slowdown slowdown NOUN bracis-19024 206 5 for for ADP bracis-19024 206 6 the the DET bracis-19024 206 7 various various ADJ bracis-19024 206 8 scenarios scenario NOUN bracis-19024 206 9 considered consider VERB bracis-19024 206 10 . . PUNCT bracis-19024 207 1 each each DET bracis-19024 207 2 bar bar NOUN bracis-19024 207 3 represents represent VERB bracis-19024 207 4 a a DET bracis-19024 207 5 different different ADJ bracis-19024 207 6 instantiation instantiation NOUN bracis-19024 207 7 of of ADP bracis-19024 207 8 the the DET bracis-19024 207 9 ( ( PUNCT bracis-19024 207 10 semi-)mdps semi-)mdps NOUN bracis-19024 207 11 . . PROPN bracis-19024 207 12 average average ADJ bracis-19024 207 13 slowdowns slowdown NOUN bracis-19024 207 14 were be AUX bracis-19024 207 15 computed compute VERB bracis-19024 207 16 by by ADP bracis-19024 207 17 averaging average VERB bracis-19024 207 18 the the DET bracis-19024 207 19 slowdown slowdown NOUN bracis-19024 207 20 of of ADP bracis-19024 207 21 a a DET bracis-19024 207 22 thousand thousand NUM bracis-19024 207 23 independent independent ADJ bracis-19024 207 24 trials trial NOUN bracis-19024 207 25 for for ADP bracis-19024 207 26 each each DET bracis-19024 207 27 agent agent NOUN bracis-19024 207 28 in in ADP bracis-19024 207 29 each each DET bracis-19024 207 30 scenario scenario NOUN bracis-19024 207 31 . . PUNCT bracis-19024 208 1 all all DET bracis-19024 208 2 agents agent NOUN bracis-19024 208 3 were be AUX bracis-19024 208 4 evaluated evaluate VERB bracis-19024 208 5 with with ADP bracis-19024 208 6 same same ADJ bracis-19024 208 7 workload workload NOUN bracis-19024 208 8 and and CCONJ bracis-19024 208 9 random random ADJ bracis-19024 208 10 seed seed NOUN bracis-19024 208 11 . . PUNCT bracis-19024 209 1 in in ADP bracis-19024 209 2 the the DET bracis-19024 209 3 legend legend NOUN bracis-19024 209 4 , , PUNCT bracis-19024 209 5 image image NOUN bracis-19024 209 6 - - PUNCT bracis-19024 209 7 like like ADJ bracis-19024 209 8 corresponds correspond NOUN bracis-19024 209 9 to to ADP bracis-19024 209 10 the the DET bracis-19024 209 11 base base PROPN bracis-19024 209 12 mdp mdp PROPN bracis-19024 209 13 , , PUNCT bracis-19024 209 14 compact compact ADJ bracis-19024 209 15 to to ADP bracis-19024 209 16 the the DET bracis-19024 209 17 compact compact ADJ bracis-19024 209 18 representation representation NOUN bracis-19024 209 19 , , PUNCT bracis-19024 209 20 sparse sparse ADJ bracis-19024 209 21 to to ADP bracis-19024 209 22 the the DET bracis-19024 209 23 sparse sparse ADJ bracis-19024 209 24 state state NOUN bracis-19024 209 25 transitions transition NOUN bracis-19024 209 26 , , PUNCT bracis-19024 209 27 and and CCONJ bracis-19024 209 28 reduced reduce VERB bracis-19024 209 29 to to ADP bracis-19024 209 30 the the DET bracis-19024 209 31 reduced reduce VERB bracis-19024 209 32 set set NOUN bracis-19024 209 33 of of ADP bracis-19024 209 34 jobs job NOUN bracis-19024 209 35 to to PART bracis-19024 209 36 compute compute VERB bracis-19024 209 37 rewards reward NOUN bracis-19024 209 38 . . PUNCT bracis-19024 210 1 full full ADJ bracis-19024 210 2 size size NOUN bracis-19024 210 3 image image NOUN bracis-19024 210 4 table table NOUN bracis-19024 210 5 3 3 NUM bracis-19024 210 6 . . PUNCT bracis-19024 210 7 key key NOUN bracis-19024 210 8 to to ADP bracis-19024 210 9 the the DET bracis-19024 210 10 scenarios scenario NOUN bracis-19024 210 11 presented present VERB bracis-19024 210 12 in in ADP bracis-19024 210 13 fig fig NOUN bracis-19024 210 14 . . PUNCT bracis-19024 210 15     SPACE bracis-19024 211 1 4 4 NUM bracis-19024 211 2 . . PUNCT bracis-19024 211 3 procs proc NOUN bracis-19024 211 4 . . PUNCT bracis-19024 212 1 refers refer VERB bracis-19024 212 2 to to ADP bracis-19024 212 3 the the DET bracis-19024 212 4 number number NOUN bracis-19024 212 5 of of ADP bracis-19024 212 6 processors processor NOUN bracis-19024 212 7 in in ADP bracis-19024 212 8 the the DET bracis-19024 212 9 cluster cluster NOUN bracis-19024 212 10 , , PUNCT bracis-19024 212 11 max max PROPN bracis-19024 212 12 length length NOUN bracis-19024 212 13 refers refer VERB bracis-19024 212 14 to to ADP bracis-19024 212 15 the the DET bracis-19024 212 16 maximum maximum ADJ bracis-19024 212 17 job job NOUN bracis-19024 212 18 length length NOUN bracis-19024 212 19 , , PUNCT bracis-19024 212 20 and and CCONJ bracis-19024 212 21 max max PROPN bracis-19024 212 22 size size NOUN bracis-19024 212 23 refers refer VERB bracis-19024 212 24 to to ADP bracis-19024 212 25 the the DET bracis-19024 212 26 maximum maximum ADJ bracis-19024 212 27 number number NOUN bracis-19024 212 28 of of ADP bracis-19024 212 29 processors processor NOUN bracis-19024 212 30 used use VERB bracis-19024 212 31 by by ADP bracis-19024 212 32 jobs.full jobs.full PROPN bracis-19024 212 33 size size NOUN bracis-19024 212 34 table table NOUN bracis-19024 212 35 when when SCONJ bracis-19024 212 36 evaluating evaluate VERB bracis-19024 212 37 agents agent NOUN bracis-19024 212 38 after after ADP bracis-19024 212 39 one one NUM bracis-19024 212 40 million million NUM bracis-19024 212 41 iterations iteration NOUN bracis-19024 212 42 , , PUNCT bracis-19024 212 43 scheduling scheduling NOUN bracis-19024 212 44 performance performance NOUN bracis-19024 212 45 was be AUX bracis-19024 212 46 similar similar ADJ bracis-19024 212 47 between between ADP bracis-19024 212 48 agents agent NOUN bracis-19024 212 49 when when SCONJ bracis-19024 212 50 the the DET bracis-19024 212 51 maximum maximum ADJ bracis-19024 212 52 number number NOUN bracis-19024 212 53 of of ADP bracis-19024 212 54 processors processor NOUN bracis-19024 212 55 used use VERB bracis-19024 212 56 by by ADP bracis-19024 212 57 jobs job NOUN bracis-19024 212 58 was be AUX bracis-19024 212 59 smaller small ADJ bracis-19024 212 60 ( ( PUNCT bracis-19024 212 61 which which PRON bracis-19024 212 62 implies imply VERB bracis-19024 212 63 less less ADJ bracis-19024 212 64 parallelism parallelism NOUN bracis-19024 212 65 ) ) PUNCT bracis-19024 212 66 . . PUNCT bracis-19024 213 1 given give VERB bracis-19024 213 2 job job NOUN bracis-19024 213 3 submission submission NOUN bracis-19024 213 4 rates rate NOUN bracis-19024 213 5 in in ADP bracis-19024 213 6 all all DET bracis-19024 213 7 environments environment NOUN bracis-19024 213 8 was be AUX bracis-19024 213 9 the the DET bracis-19024 213 10 same same ADJ bracis-19024 213 11 , , PUNCT bracis-19024 213 12 clusters cluster NOUN bracis-19024 213 13 were be AUX bracis-19024 213 14 less less ADV bracis-19024 213 15 busy busy ADJ bracis-19024 213 16 in in ADP bracis-19024 213 17 these these DET bracis-19024 213 18 situations situation NOUN bracis-19024 213 19 : : PUNCT bracis-19024 213 20 as as ADV bracis-19024 213 21 long long ADV bracis-19024 213 22 as as SCONJ bracis-19024 213 23 jobs job NOUN bracis-19024 213 24 are be AUX bracis-19024 213 25 scheduled schedule VERB bracis-19024 213 26 , , PUNCT bracis-19024 213 27 there there PRON bracis-19024 213 28 should should AUX bracis-19024 213 29 n’t not PART bracis-19024 213 30 be be AUX bracis-19024 213 31 significant significant ADJ bracis-19024 213 32 differences difference NOUN bracis-19024 213 33 in in ADP bracis-19024 213 34 average average ADJ bracis-19024 213 35 slowdown slowdown NOUN bracis-19024 213 36 , , PUNCT bracis-19024 213 37 due due ADP bracis-19024 213 38 to to ADP bracis-19024 213 39 smaller small ADJ bracis-19024 213 40 queues queue NOUN bracis-19024 213 41 . . PUNCT bracis-19024 214 1 in in ADP bracis-19024 214 2 fig fig NOUN bracis-19024 214 3 . . PUNCT bracis-19024 214 4     SPACE bracis-19024 214 5 4 4 NUM bracis-19024 214 6 , , PUNCT bracis-19024 214 7 with with ADP bracis-19024 214 8 key key NOUN bracis-19024 214 9 to to ADP bracis-19024 214 10 scenarions scenarion NOUN bracis-19024 214 11 shown show VERB bracis-19024 214 12 in in ADP bracis-19024 214 13 table table NOUN bracis-19024 214 14     SPACE bracis-19024 214 15 3 3 NUM bracis-19024 214 16 , , PUNCT bracis-19024 214 17 we we PRON bracis-19024 214 18 show show VERB bracis-19024 214 19 average average ADJ bracis-19024 214 20 slowdown slowdown NOUN bracis-19024 214 21 of of ADP bracis-19024 214 22 the the DET bracis-19024 214 23 agents agent NOUN bracis-19024 214 24 for for ADP bracis-19024 214 25 the the DET bracis-19024 214 26 scenarios scenario NOUN bracis-19024 214 27 in in ADP bracis-19024 214 28 which which PRON bracis-19024 214 29 there there PRON bracis-19024 214 30 was be VERB bracis-19024 214 31 some some DET bracis-19024 214 32 variability variability NOUN bracis-19024 214 33 in in ADP bracis-19024 214 34 performance performance NOUN bracis-19024 214 35 between between ADP bracis-19024 214 36 agents agent NOUN bracis-19024 214 37 . . PUNCT bracis-19024 215 1 from from ADP bracis-19024 215 2 the the DET bracis-19024 215 3 figure figure NOUN bracis-19024 215 4 , , PUNCT bracis-19024 215 5 we we PRON bracis-19024 215 6 see see VERB bracis-19024 215 7 that that SCONJ bracis-19024 215 8 , , PUNCT bracis-19024 215 9 apart apart ADV bracis-19024 215 10 from from ADP bracis-19024 215 11 scenarios scenario NOUN bracis-19024 215 12 2 2 NUM bracis-19024 215 13 and and CCONJ bracis-19024 215 14 6 6 NUM bracis-19024 215 15 , , PUNCT bracis-19024 215 16 agent agent NOUN bracis-19024 215 17 performance performance NOUN bracis-19024 215 18 in in ADP bracis-19024 215 19 the the DET bracis-19024 215 20 “ " PUNCT bracis-19024 215 21 compact compact ADJ bracis-19024 215 22 + + CCONJ bracis-19024 215 23 sparse sparse ADJ bracis-19024 215 24 + + CCONJ bracis-19024 215 25 reduced reduced ADJ bracis-19024 215 26 ” " PUNCT bracis-19024 215 27 mdp mdp NOUN bracis-19024 215 28 is be AUX bracis-19024 215 29 not not PART bracis-19024 215 30 worse bad ADJ bracis-19024 215 31 than than ADP bracis-19024 215 32 that that PRON bracis-19024 215 33 of of ADP bracis-19024 215 34 the the DET bracis-19024 215 35 image image NOUN bracis-19024 215 36 - - PUNCT bracis-19024 215 37 like like ADJ bracis-19024 215 38 mdp mdp NOUN bracis-19024 215 39 . . PUNCT bracis-19024 216 1 of of ADP bracis-19024 216 2 these these DET bracis-19024 216 3 two two NUM bracis-19024 216 4 , , PUNCT bracis-19024 216 5 only only ADV bracis-19024 216 6 the the DET bracis-19024 216 7 difference difference NOUN bracis-19024 216 8 for for ADP bracis-19024 216 9 scenario scenario NOUN bracis-19024 216 10 6 6 NUM bracis-19024 216 11 is be AUX bracis-19024 216 12 statistically statistically ADV bracis-19024 216 13 significant significant ADJ bracis-19024 216 14 , , PUNCT bracis-19024 216 15 with with ADP bracis-19024 216 16 p p NOUN bracis-19024 216 17 - - PUNCT bracis-19024 216 18 value value NOUN bracis-19024 216 19 \(\le \(\le PROPN bracis-19024 216 20 5\%\ 5\%\ NUM bracis-19024 216 21 ) ) PUNCT bracis-19024 216 22 when when SCONJ bracis-19024 216 23 performing perform VERB bracis-19024 216 24 a a DET bracis-19024 216 25 t t NOUN bracis-19024 216 26 - - PUNCT bracis-19024 216 27 test test NOUN bracis-19024 216 28 with with ADP bracis-19024 216 29 alternative alternative ADJ bracis-19024 216 30 hypothesis hypothesis NOUN bracis-19024 216 31 of of ADP bracis-19024 216 32 different different ADJ bracis-19024 216 33 distributions distribution NOUN bracis-19024 216 34 . . PUNCT bracis-19024 217 1 for for ADP bracis-19024 217 2 the the DET bracis-19024 217 3 cases case NOUN bracis-19024 217 4 where where SCONJ bracis-19024 217 5 “ " PUNCT bracis-19024 217 6 compact compact ADJ bracis-19024 217 7 + + CCONJ bracis-19024 217 8 sparse sparse ADJ bracis-19024 217 9 + + CCONJ bracis-19024 217 10 reduced reduced ADJ bracis-19024 217 11 ” " PUNCT bracis-19024 217 12 agents agent NOUN bracis-19024 217 13 are be AUX bracis-19024 217 14 better well ADJ bracis-19024 217 15 , , PUNCT bracis-19024 217 16 the the DET bracis-19024 217 17 results result NOUN bracis-19024 217 18 are be AUX bracis-19024 217 19 statistically statistically ADV bracis-19024 217 20 significant significant ADJ bracis-19024 217 21 ( ( PUNCT bracis-19024 217 22 p p NOUN bracis-19024 217 23 - - PUNCT bracis-19024 217 24 value value NOUN bracis-19024 217 25 \(\le \(\le PROPN bracis-19024 217 26 5\%\ 5\%\ NUM bracis-19024 217 27 ) ) PUNCT bracis-19024 217 28 ) ) PUNCT bracis-19024 217 29 in in ADP bracis-19024 217 30 scenarios scenario NOUN bracis-19024 217 31 3 3 NUM bracis-19024 217 32 , , PUNCT bracis-19024 217 33 4 4 NUM bracis-19024 217 34 , , PUNCT bracis-19024 217 35 5 5 NUM bracis-19024 217 36 , , PUNCT bracis-19024 217 37 7 7 NUM bracis-19024 217 38 , , PUNCT bracis-19024 217 39 and and CCONJ bracis-19024 217 40 9 9 NUM bracis-19024 217 41 . . PUNCT bracis-19024 218 1 scenarios scenario NOUN bracis-19024 218 2 2 2 NUM bracis-19024 218 3 and and CCONJ bracis-19024 218 4 6 6 NUM bracis-19024 218 5 are be AUX bracis-19024 218 6 interesting interesting ADJ bracis-19024 218 7 , , PUNCT bracis-19024 218 8 since since SCONJ bracis-19024 218 9 they they PRON bracis-19024 218 10 were be AUX bracis-19024 218 11 configured configure VERB bracis-19024 218 12 to to PART bracis-19024 218 13 have have VERB bracis-19024 218 14 shorter short ADJ bracis-19024 218 15 jobs job NOUN bracis-19024 218 16 of of ADP bracis-19024 218 17 at at ADP bracis-19024 218 18 most most ADJ bracis-19024 218 19 15 15 NUM bracis-19024 218 20 time time NOUN bracis-19024 218 21 - - PUNCT bracis-19024 218 22 steps step NOUN bracis-19024 218 23 , , PUNCT bracis-19024 218 24 with with ADP bracis-19024 218 25 scenario scenario NOUN bracis-19024 218 26 1 1 NUM bracis-19024 218 27 having have VERB bracis-19024 218 28 10 10 NUM bracis-19024 218 29 processors processor NOUN bracis-19024 218 30 , , PUNCT bracis-19024 218 31 and and CCONJ bracis-19024 218 32 scenario scenario NOUN bracis-19024 218 33 6 6 NUM bracis-19024 218 34 having have VERB bracis-19024 218 35 64 64 NUM bracis-19024 218 36 , , PUNCT bracis-19024 218 37 both both CCONJ bracis-19024 218 38 with with ADP bracis-19024 218 39 jobs job NOUN bracis-19024 218 40 with with ADP bracis-19024 218 41 the the DET bracis-19024 218 42 potential potential NOUN bracis-19024 218 43 of of ADP bracis-19024 218 44 using use VERB bracis-19024 218 45 all all DET bracis-19024 218 46 cluster cluster NOUN bracis-19024 218 47 resources resource NOUN bracis-19024 218 48 . . PUNCT bracis-19024 219 1 fig fig NOUN bracis-19024 219 2 . . PUNCT bracis-19024 220 1 5 5 NUM bracis-19024 220 2 . . NOUN bracis-19024 220 3 time time NOUN bracis-19024 220 4 needed need VERB bracis-19024 220 5 to to PART bracis-19024 220 6 train train VERB bracis-19024 220 7 agents agent NOUN bracis-19024 220 8 for for ADP bracis-19024 220 9 three three NUM bracis-19024 220 10 million million NUM bracis-19024 220 11 iterations iteration NOUN bracis-19024 220 12 . . PUNCT bracis-19024 221 1 the the DET bracis-19024 221 2 shaded shaded ADJ bracis-19024 221 3 area area NOUN bracis-19024 221 4 represents represent VERB bracis-19024 221 5 one one NUM bracis-19024 221 6 standard standard ADJ bracis-19024 221 7 deviation deviation NOUN bracis-19024 221 8 . . PUNCT bracis-19024 222 1 increasing increase VERB bracis-19024 222 2 the the DET bracis-19024 222 3 time time NOUN bracis-19024 222 4 horizon horizon NOUN bracis-19024 222 5 increases increase VERB bracis-19024 222 6 the the DET bracis-19024 222 7 training training NOUN bracis-19024 222 8 time time NOUN bracis-19024 222 9 of of ADP bracis-19024 222 10 compact compact ADJ bracis-19024 222 11 agents agent NOUN bracis-19024 222 12 by by ADP bracis-19024 222 13 a a DET bracis-19024 222 14 constant constant ADJ bracis-19024 222 15 factor factor NOUN bracis-19024 222 16 , , PUNCT bracis-19024 222 17 while while SCONJ bracis-19024 222 18 it it PRON bracis-19024 222 19 adds add VERB bracis-19024 222 20 a a DET bracis-19024 222 21 linear linear ADJ bracis-19024 222 22 factor factor NOUN bracis-19024 222 23 to to ADP bracis-19024 222 24 the the DET bracis-19024 222 25 training training NOUN bracis-19024 222 26 times time NOUN bracis-19024 222 27 of of ADP bracis-19024 222 28 agents agent NOUN bracis-19024 222 29 that that PRON bracis-19024 222 30 use use VERB bracis-19024 222 31 an an DET bracis-19024 222 32 image image NOUN bracis-19024 222 33 - - PUNCT bracis-19024 222 34 like like ADJ bracis-19024 222 35 representation representation NOUN bracis-19024 222 36 . . PUNCT bracis-19024 223 1 full full ADJ bracis-19024 223 2 size size NOUN bracis-19024 223 3 image image NOUN bracis-19024 223 4 in in ADP bracis-19024 223 5 fig fig NOUN bracis-19024 223 6 . . PUNCT bracis-19024 223 7     SPACE bracis-19024 224 1 5 5 NUM bracis-19024 224 2 we we PRON bracis-19024 224 3 contrast contrast VERB bracis-19024 224 4 the the DET bracis-19024 224 5 training training NOUN bracis-19024 224 6 times time NOUN bracis-19024 224 7 for for ADP bracis-19024 224 8 the the DET bracis-19024 224 9 various various ADJ bracis-19024 224 10 agents agent NOUN bracis-19024 224 11 . . PUNCT bracis-19024 225 1 as as SCONJ bracis-19024 225 2 can can AUX bracis-19024 225 3 be be AUX bracis-19024 225 4 seen see VERB bracis-19024 225 5 , , PUNCT bracis-19024 225 6 training training NOUN bracis-19024 225 7 times time NOUN bracis-19024 225 8 for for ADP bracis-19024 225 9 agents agent NOUN bracis-19024 225 10 based base VERB bracis-19024 225 11 on on ADP bracis-19024 225 12 the the DET bracis-19024 225 13 image image NOUN bracis-19024 225 14 - - PUNCT bracis-19024 225 15 like like ADJ bracis-19024 225 16 mdp mdp NOUN bracis-19024 225 17 are be AUX bracis-19024 225 18 highly highly ADV bracis-19024 225 19 variable variable ADJ bracis-19024 225 20 , , PUNCT bracis-19024 225 21 due due ADP bracis-19024 225 22 to to ADP bracis-19024 225 23 the the DET bracis-19024 225 24 fact fact NOUN bracis-19024 225 25 that that SCONJ bracis-19024 225 26 different different ADJ bracis-19024 225 27 mdp mdp PROPN bracis-19024 225 28 configurations configuration NOUN bracis-19024 225 29 result result VERB bracis-19024 225 30 in in ADP bracis-19024 225 31 different different ADJ bracis-19024 225 32 sizes size NOUN bracis-19024 225 33 of of ADP bracis-19024 225 34 state state NOUN bracis-19024 225 35 representations representation NOUN bracis-19024 225 36 , , PUNCT bracis-19024 225 37 which which PRON bracis-19024 225 38 impacts impact VERB bracis-19024 225 39 training training NOUN bracis-19024 225 40 performance performance NOUN bracis-19024 225 41 . . PUNCT bracis-19024 226 1 as as ADP bracis-19024 226 2 an an DET bracis-19024 226 3 example example NOUN bracis-19024 226 4 , , PUNCT bracis-19024 226 5 the the DET bracis-19024 226 6 image image NOUN bracis-19024 226 7 - - PUNCT bracis-19024 226 8 like like ADJ bracis-19024 226 9 agent agent NOUN bracis-19024 226 10 requires require VERB bracis-19024 226 11 301068 301068 NUM bracis-19024 226 12 , , PUNCT bracis-19024 226 13 1089548 1089548 NUM bracis-19024 226 14 , , PUNCT bracis-19024 226 15 and and CCONJ bracis-19024 226 16 1821708 1821708 NUM bracis-19024 226 17 parameters parameter NOUN bracis-19024 226 18 for for ADP bracis-19024 226 19 the the DET bracis-19024 226 20 scenarios scenario NOUN bracis-19024 226 21 with with ADP bracis-19024 226 22 10 10 NUM bracis-19024 226 23 , , PUNCT bracis-19024 226 24 38 38 NUM bracis-19024 226 25 , , PUNCT bracis-19024 226 26 and and CCONJ bracis-19024 226 27 64 64 NUM bracis-19024 226 28 processors processor NOUN bracis-19024 226 29 , , PUNCT bracis-19024 226 30 while while SCONJ bracis-19024 226 31 all all DET bracis-19024 226 32 compact compact ADJ bracis-19024 226 33 agents agent NOUN bracis-19024 226 34 require require VERB bracis-19024 226 35 a a DET bracis-19024 226 36 fixed fix VERB bracis-19024 226 37 number number NOUN bracis-19024 226 38 of of ADP bracis-19024 226 39 parameters parameter NOUN bracis-19024 226 40 : : PUNCT bracis-19024 226 41 24332 24332 NUM bracis-19024 226 42 . . PUNCT bracis-19024 227 1 times time NOUN bracis-19024 227 2 were be AUX bracis-19024 227 3 measured measure VERB bracis-19024 227 4 in in ADP bracis-19024 227 5 a a DET bracis-19024 227 6 linux linux PROPN bracis-19024 227 7 5.10.42 5.10.42 NUM bracis-19024 227 8 desktop desktop NOUN bracis-19024 227 9 with with ADP bracis-19024 227 10 an an DET bracis-19024 227 11 nvidia nvidia PROPN bracis-19024 227 12 gtx gtx PROPN bracis-19024 227 13 1070 1070 NUM bracis-19024 227 14 gpu gpu X bracis-19024 227 15 and and CCONJ bracis-19024 227 16 an an DET bracis-19024 227 17 i7–8700k i7–8700k NOUN bracis-19024 227 18 processor processor NOUN bracis-19024 227 19 using use VERB bracis-19024 227 20 the the DET bracis-19024 227 21 performance performance NOUN bracis-19024 227 22 cpu cpu NOUN bracis-19024 227 23 frequency frequency NOUN bracis-19024 227 24 - - PUNCT bracis-19024 227 25 scaling scale VERB bracis-19024 227 26 governor governor NOUN bracis-19024 227 27 . . PUNCT bracis-19024 228 1 the the DET bracis-19024 228 2 compact compact ADJ bracis-19024 228 3 mdps mdps NOUN bracis-19024 228 4 proposed propose VERB bracis-19024 228 5 in in ADP bracis-19024 228 6 this this DET bracis-19024 228 7 paper paper NOUN bracis-19024 228 8 all all PRON bracis-19024 228 9 have have VERB bracis-19024 228 10 the the DET bracis-19024 228 11 characteristic characteristic NOUN bracis-19024 228 12 of of ADP bracis-19024 228 13 having have VERB bracis-19024 228 14 a a DET bracis-19024 228 15 state state NOUN bracis-19024 228 16 representation representation NOUN bracis-19024 228 17 with with ADP bracis-19024 228 18 a a DET bracis-19024 228 19 fixed fix VERB bracis-19024 228 20 size size NOUN bracis-19024 228 21 , , PUNCT bracis-19024 228 22 which which PRON bracis-19024 228 23 allows allow VERB bracis-19024 228 24 for for ADP bracis-19024 228 25 transfer transfer NOUN bracis-19024 228 26 of of ADP bracis-19024 228 27 learned learn VERB bracis-19024 228 28 weights weight NOUN bracis-19024 228 29 between between ADP bracis-19024 228 30 mdps mdps PROPN bracis-19024 228 31 . . PUNCT bracis-19024 229 1 here here ADV bracis-19024 229 2 , , PUNCT bracis-19024 229 3 we we PRON bracis-19024 229 4 consider consider VERB bracis-19024 229 5 transfer transfer VERB bracis-19024 229 6 the the DET bracis-19024 229 7 ability ability NOUN bracis-19024 229 8 to to PART bracis-19024 229 9 change change VERB bracis-19024 229 10 cluster cluster NOUN bracis-19024 229 11 configuration configuration NOUN bracis-19024 229 12 without without ADP bracis-19024 229 13 the the DET bracis-19024 229 14 need need NOUN bracis-19024 229 15 for for ADP bracis-19024 229 16 retraining retrain VERB bracis-19024 229 17 an an DET bracis-19024 229 18 agent agent NOUN bracis-19024 229 19 from from ADP bracis-19024 229 20 scratch scratch NOUN bracis-19024 229 21 , , PUNCT bracis-19024 229 22 which which PRON bracis-19024 229 23 is be AUX bracis-19024 229 24 simply simply ADV bracis-19024 229 25 not not PART bracis-19024 229 26 possible possible ADJ bracis-19024 229 27 when when SCONJ bracis-19024 229 28 using use VERB bracis-19024 229 29 the the DET bracis-19024 229 30 image image NOUN bracis-19024 229 31 - - PUNCT bracis-19024 229 32 like like ADJ bracis-19024 229 33 representation representation NOUN bracis-19024 229 34 . . PUNCT bracis-19024 230 1 in in ADP bracis-19024 230 2 fig fig NOUN bracis-19024 230 3 . . PUNCT bracis-19024 230 4     SPACE bracis-19024 230 5 6 6 NUM bracis-19024 230 6 , , PUNCT bracis-19024 230 7 for for ADP bracis-19024 230 8 example example NOUN bracis-19024 230 9 , , PUNCT bracis-19024 230 10 we we PRON bracis-19024 230 11 show show VERB bracis-19024 230 12 the the DET bracis-19024 230 13 performance performance NOUN bracis-19024 230 14 of of ADP bracis-19024 230 15 an an DET bracis-19024 230 16 agent agent NOUN bracis-19024 230 17 trained train VERB bracis-19024 230 18 in in ADP bracis-19024 230 19 the the DET bracis-19024 230 20 bounded bounded ADJ bracis-19024 230 21 reward reward NOUN bracis-19024 230 22 , , PUNCT bracis-19024 230 23 event event NOUN bracis-19024 230 24 - - PUNCT bracis-19024 230 25 based base VERB bracis-19024 230 26 , , PUNCT bracis-19024 230 27 compact compact ADJ bracis-19024 230 28 mdp mdp NOUN bracis-19024 230 29 with with ADP bracis-19024 230 30 64 64 NUM bracis-19024 230 31 processors processor NOUN bracis-19024 230 32 and and CCONJ bracis-19024 230 33 with with ADP bracis-19024 230 34 jobs job NOUN bracis-19024 230 35 of of ADP bracis-19024 230 36 length length NOUN bracis-19024 230 37 33 33 NUM bracis-19024 230 38 ( ( PUNCT bracis-19024 230 39 the the DET bracis-19024 230 40 best good ADJ bracis-19024 230 41 agent agent NOUN bracis-19024 230 42 in in ADP bracis-19024 230 43 fig fig NOUN bracis-19024 230 44 . . PUNCT bracis-19024 230 45     SPACE bracis-19024 231 1 4 4 NUM bracis-19024 231 2 , , PUNCT bracis-19024 231 3 corresponding correspond VERB bracis-19024 231 4 to to ADP bracis-19024 231 5 scenario scenario NOUN bracis-19024 231 6 9 9 NUM bracis-19024 231 7 ) ) PUNCT bracis-19024 231 8 evaluated evaluate VERB bracis-19024 231 9 in in ADP bracis-19024 231 10 a a DET bracis-19024 231 11 compact compact ADJ bracis-19024 231 12 environment environment NOUN bracis-19024 231 13 without without ADP bracis-19024 231 14 event event NOUN bracis-19024 231 15 - - PUNCT bracis-19024 231 16 based base VERB bracis-19024 231 17 updates update NOUN bracis-19024 231 18 . . PUNCT bracis-19024 232 1 with with ADP bracis-19024 232 2 this this DET bracis-19024 232 3 same same ADJ bracis-19024 232 4 agent agent NOUN bracis-19024 232 5 , , PUNCT bracis-19024 232 6 we we PRON bracis-19024 232 7 were be AUX bracis-19024 232 8 able able ADJ bracis-19024 232 9 to to PART bracis-19024 232 10 evaluate evaluate VERB bracis-19024 232 11 its its PRON bracis-19024 232 12 performance performance NOUN bracis-19024 232 13 in in ADP bracis-19024 232 14 all all DET bracis-19024 232 15 different different ADJ bracis-19024 232 16 scenarios scenario NOUN bracis-19024 232 17 , , PUNCT bracis-19024 232 18 without without ADP bracis-19024 232 19 the the DET bracis-19024 232 20 need need NOUN bracis-19024 232 21 for for ADP bracis-19024 232 22 retraining retrain VERB bracis-19024 232 23 . . PUNCT bracis-19024 233 1 we we PRON bracis-19024 233 2 see see VERB bracis-19024 233 3 that that SCONJ bracis-19024 233 4 , , PUNCT bracis-19024 233 5 for for ADP bracis-19024 233 6 the the DET bracis-19024 233 7 most most ADJ bracis-19024 233 8 part part NOUN bracis-19024 233 9 , , PUNCT bracis-19024 233 10 slowdown slowdown NOUN bracis-19024 233 11 is be AUX bracis-19024 233 12 kept keep VERB bracis-19024 233 13 low low ADJ bracis-19024 233 14 , , PUNCT bracis-19024 233 15 and and CCONJ bracis-19024 233 16 not not PART bracis-19024 233 17 only only ADV bracis-19024 233 18 that that SCONJ bracis-19024 233 19 : : PUNCT bracis-19024 233 20 this this DET bracis-19024 233 21 agent agent NOUN bracis-19024 233 22 outperformed outperform VERB bracis-19024 233 23 other other ADJ bracis-19024 233 24 agents agent NOUN bracis-19024 233 25 in in ADP bracis-19024 233 26 80 80 NUM bracis-19024 233 27 % % NOUN bracis-19024 233 28 of of ADP bracis-19024 233 29 scenarios scenario NOUN bracis-19024 233 30 ( ( PUNCT bracis-19024 233 31 differences difference NOUN bracis-19024 233 32 are be AUX bracis-19024 233 33 statistically statistically ADV bracis-19024 233 34 significant significant ADJ bracis-19024 233 35 , , PUNCT bracis-19024 233 36 with with ADP bracis-19024 233 37 p p NOUN bracis-19024 233 38 - - PUNCT bracis-19024 233 39 value value NOUN bracis-19024 233 40 \(\le \(\le PROPN bracis-19024 233 41 1\%\ 1\%\ NUM bracis-19024 233 42 ) ) PUNCT bracis-19024 233 43 , , PUNCT bracis-19024 233 44 except except SCONJ bracis-19024 233 45 for for ADP bracis-19024 233 46 scenario scenario NOUN bracis-19024 233 47 9 9 NUM bracis-19024 233 48 , , PUNCT bracis-19024 233 49 since since SCONJ bracis-19024 233 50 this this PRON bracis-19024 233 51 is be AUX bracis-19024 233 52 the the DET bracis-19024 233 53 same same ADJ bracis-19024 233 54 agent agent NOUN bracis-19024 233 55 , , PUNCT bracis-19024 233 56 and and CCONJ bracis-19024 233 57 scenario scenario NOUN bracis-19024 233 58 5 5 NUM bracis-19024 233 59 , , PUNCT bracis-19024 233 60 where where SCONJ bracis-19024 233 61 the the DET bracis-19024 233 62 test test NOUN bracis-19024 233 63 has have VERB bracis-19024 233 64 low low ADJ bracis-19024 233 65 power power NOUN bracis-19024 233 66 to to PART bracis-19024 233 67 reject reject VERB bracis-19024 233 68 the the DET bracis-19024 233 69 null null ADJ bracis-19024 233 70 hypothesis hypothesis NOUN bracis-19024 233 71 ) ) PUNCT bracis-19024 233 72 . . PUNCT bracis-19024 234 1 this this PRON bracis-19024 234 2 highlights highlight VERB bracis-19024 234 3 the the DET bracis-19024 234 4 advantage advantage NOUN bracis-19024 234 5 of of ADP bracis-19024 234 6 using use VERB bracis-19024 234 7 a a DET bracis-19024 234 8 representation representation NOUN bracis-19024 234 9 that that PRON bracis-19024 234 10 allows allow VERB bracis-19024 234 11 for for ADP bracis-19024 234 12 easy easy ADJ bracis-19024 234 13 transfer transfer NOUN bracis-19024 234 14 between between ADP bracis-19024 234 15 agents agent NOUN bracis-19024 234 16 , , PUNCT bracis-19024 234 17 enabling enable VERB bracis-19024 234 18 good good ADJ bracis-19024 234 19 performance performance NOUN bracis-19024 234 20 in in ADP bracis-19024 234 21 a a DET bracis-19024 234 22 variety variety NOUN bracis-19024 234 23 of of ADP bracis-19024 234 24 cluster cluster NOUN bracis-19024 234 25 settings setting NOUN bracis-19024 234 26 . . PUNCT bracis-19024 235 1 fig fig NOUN bracis-19024 235 2 . . PUNCT bracis-19024 236 1 6 6 NUM bracis-19024 236 2 . . X bracis-19024 236 3 bar bar NOUN bracis-19024 236 4 chart chart NOUN bracis-19024 236 5 contrasting contrast VERB bracis-19024 236 6 the the DET bracis-19024 236 7 performance performance NOUN bracis-19024 236 8 of of ADP bracis-19024 236 9 a a DET bracis-19024 236 10 transferred transfer VERB bracis-19024 236 11 agent agent NOUN bracis-19024 236 12 to to ADP bracis-19024 236 13 agents agent NOUN bracis-19024 236 14 trained train VERB bracis-19024 236 15 specifically specifically ADV bracis-19024 236 16 in in ADP bracis-19024 236 17 their their PRON bracis-19024 236 18 environments environment NOUN bracis-19024 236 19 . . PUNCT bracis-19024 237 1 full full ADJ bracis-19024 237 2 size size NOUN bracis-19024 237 3 image image NOUN bracis-19024 237 4 6 6 NUM bracis-19024 237 5 conclusion conclusion NOUN bracis-19024 237 6 in in ADP bracis-19024 237 7 this this DET bracis-19024 237 8 paper paper NOUN bracis-19024 237 9 , , PUNCT bracis-19024 237 10 we we PRON bracis-19024 237 11 ’ve ’ve AUX bracis-19024 237 12 filled fill VERB bracis-19024 237 13 a a DET bracis-19024 237 14 gap gap NOUN bracis-19024 237 15 in in ADP bracis-19024 237 16 the the DET bracis-19024 237 17 literature literature NOUN bracis-19024 237 18 by by ADP bracis-19024 237 19 analyzing analyze VERB bracis-19024 237 20 the the DET bracis-19024 237 21 effects effect NOUN bracis-19024 237 22 of of ADP bracis-19024 237 23 different different ADJ bracis-19024 237 24 mdp mdp NOUN bracis-19024 237 25 design design NOUN bracis-19024 237 26 decisions decision NOUN bracis-19024 237 27 on on ADP bracis-19024 237 28 the the DET bracis-19024 237 29 behavior behavior NOUN bracis-19024 237 30 of of ADP bracis-19024 237 31 rl rl NOUN bracis-19024 237 32 agents agent NOUN bracis-19024 237 33 . . PUNCT bracis-19024 238 1 in in ADP bracis-19024 238 2 particular particular ADJ bracis-19024 238 3 , , PUNCT bracis-19024 238 4 we we PRON bracis-19024 238 5 experimented experiment VERB bracis-19024 238 6 with with ADP bracis-19024 238 7 resource resource NOUN bracis-19024 238 8 management management NOUN bracis-19024 238 9 agents agent NOUN bracis-19024 238 10 for for ADP bracis-19024 238 11 job job NOUN bracis-19024 238 12 scheduling scheduling NOUN bracis-19024 238 13 in in ADP bracis-19024 238 14 computing computing NOUN bracis-19024 238 15 clusters cluster NOUN bracis-19024 238 16 , , PUNCT bracis-19024 238 17 discussing discuss VERB bracis-19024 238 18 cases case NOUN bracis-19024 238 19 in in ADP bracis-19024 238 20 which which PRON bracis-19024 238 21 a a DET bracis-19024 238 22 compact compact ADJ bracis-19024 238 23 representation representation NOUN bracis-19024 238 24 outperforms outperform VERB bracis-19024 238 25 a a DET bracis-19024 238 26 dense dense ADJ bracis-19024 238 27 one one NOUN bracis-19024 238 28 , , PUNCT bracis-19024 238 29 and and CCONJ bracis-19024 238 30 vice vice NOUN bracis-19024 238 31 - - NOUN bracis-19024 238 32 versa versa ADJ bracis-19024 238 33 . . PUNCT bracis-19024 239 1 we we PRON bracis-19024 239 2 proposed propose VERB bracis-19024 239 3 a a DET bracis-19024 239 4 new new ADJ bracis-19024 239 5 state state NOUN bracis-19024 239 6 representation representation NOUN bracis-19024 239 7 , , PUNCT bracis-19024 239 8 a a DET bracis-19024 239 9 transition transition NOUN bracis-19024 239 10 function function NOUN bracis-19024 239 11 , , PUNCT bracis-19024 239 12 and and CCONJ bracis-19024 239 13 a a DET bracis-19024 239 14 reward reward NOUN bracis-19024 239 15 function function NOUN bracis-19024 239 16 for for ADP bracis-19024 239 17 an an DET bracis-19024 239 18 mdp mdp NOUN bracis-19024 239 19 studied study VERB bracis-19024 239 20 in in ADP bracis-19024 239 21 the the DET bracis-19024 239 22 literature literature NOUN bracis-19024 239 23 , , PUNCT bracis-19024 239 24 and and CCONJ bracis-19024 239 25 we we PRON bracis-19024 239 26 saw see VERB bracis-19024 239 27 that that SCONJ bracis-19024 239 28 these these DET bracis-19024 239 29 environments environment NOUN bracis-19024 239 30 support support VERB bracis-19024 239 31 transferring transfer VERB bracis-19024 239 32 agents agent NOUN bracis-19024 239 33 between between ADP bracis-19024 239 34 different different ADJ bracis-19024 239 35 cluster cluster NOUN bracis-19024 239 36 settings setting NOUN bracis-19024 239 37 , , PUNCT bracis-19024 239 38 while while SCONJ bracis-19024 239 39 also also ADV bracis-19024 239 40 keeping keep VERB bracis-19024 239 41 agent agent NOUN bracis-19024 239 42 memory memory NOUN bracis-19024 239 43 consumption consumption NOUN bracis-19024 239 44 constant constant ADJ bracis-19024 239 45 , , PUNCT bracis-19024 239 46 and and CCONJ bracis-19024 239 47 processing processing NOUN bracis-19024 239 48 requirements requirement NOUN bracis-19024 239 49 stable stable ADJ bracis-19024 239 50 . . PUNCT bracis-19024 240 1 we we PRON bracis-19024 240 2 also also ADV bracis-19024 240 3 saw see VERB bracis-19024 240 4 that that SCONJ bracis-19024 240 5 these these DET bracis-19024 240 6 compact compact ADJ bracis-19024 240 7 representations representation NOUN bracis-19024 240 8 are be AUX bracis-19024 240 9 no no ADV bracis-19024 240 10 worse bad ADJ bracis-19024 240 11 than than ADP bracis-19024 240 12 image image NOUN bracis-19024 240 13 - - PUNCT bracis-19024 240 14 like like ADJ bracis-19024 240 15 ones one NOUN bracis-19024 240 16 , , PUNCT bracis-19024 240 17 and and CCONJ bracis-19024 240 18 , , PUNCT bracis-19024 240 19 thus thus ADV bracis-19024 240 20 , , PUNCT bracis-19024 240 21 might might AUX bracis-19024 240 22 be be AUX bracis-19024 240 23 preferable preferable ADJ bracis-19024 240 24 when when SCONJ bracis-19024 240 25 constant constant ADJ bracis-19024 240 26 memory memory NOUN bracis-19024 240 27 usage usage NOUN bracis-19024 240 28 is be AUX bracis-19024 240 29 a a DET bracis-19024 240 30 requirement requirement NOUN bracis-19024 240 31 . . PUNCT bracis-19024 241 1 moreover moreover ADV bracis-19024 241 2 , , PUNCT bracis-19024 241 3 our our PRON bracis-19024 241 4 results result NOUN bracis-19024 241 5 indicate indicate VERB bracis-19024 241 6 that that SCONJ bracis-19024 241 7 transferred transfer VERB bracis-19024 241 8 agents agent NOUN bracis-19024 241 9 may may AUX bracis-19024 241 10 outperform outperform VERB bracis-19024 241 11 specialized specialized ADJ bracis-19024 241 12 agents agent NOUN bracis-19024 241 13 in in ADP bracis-19024 241 14 80 80 NUM bracis-19024 241 15 % % NOUN bracis-19024 241 16 of of ADP bracis-19024 241 17 the the DET bracis-19024 241 18 tested test VERB bracis-19024 241 19 scenarios scenario NOUN bracis-19024 241 20 without without ADP bracis-19024 241 21 the the DET bracis-19024 241 22 need need NOUN bracis-19024 241 23 for for ADP bracis-19024 241 24 retraining retrain VERB bracis-19024 241 25 . . PUNCT bracis-19024 242 1 notes note VERB bracis-19024 242 2 1.some 1.some NUM bracis-19024 242 3 schedulers scheduler NOUN bracis-19024 242 4 allow allow VERB bracis-19024 242 5 for for ADP bracis-19024 242 6 oversubscription oversubscription NOUN bracis-19024 242 7 of of ADP bracis-19024 242 8 memory memory NOUN bracis-19024 242 9 resources resource NOUN bracis-19024 242 10 in in ADP bracis-19024 242 11 their their PRON bracis-19024 242 12 default default NOUN bracis-19024 242 13 configuration configuration NOUN bracis-19024 242 14 , , PUNCT bracis-19024 242 15 inspired inspire VERB bracis-19024 242 16 by by ADP bracis-19024 242 17 the the DET bracis-19024 242 18 fact fact NOUN bracis-19024 242 19 that that SCONJ bracis-19024 242 20 jobs job NOUN bracis-19024 242 21 do do AUX bracis-19024 242 22 n’t not PART bracis-19024 242 23 use use VERB bracis-19024 242 24 peak peak NOUN bracis-19024 242 25 memory memory NOUN bracis-19024 242 26 during during ADP bracis-19024 242 27 their their PRON bracis-19024 242 28 complete complete ADJ bracis-19024 242 29 lifetimes lifetime NOUN bracis-19024 242 30 . . PUNCT bracis-19024 243 1 2.some 2.some NUM bracis-19024 243 2 authors author NOUN bracis-19024 243 3 leave leave VERB bracis-19024 243 4 the the DET bracis-19024 243 5 \(\gamma \(\gamma NOUN bracis-19024 243 6 \ \ NOUN bracis-19024 243 7 ) ) PUNCT bracis-19024 243 8 component component NOUN bracis-19024 243 9 out out ADP bracis-19024 243 10 of of ADP bracis-19024 243 11 the the DET bracis-19024 243 12 definition definition NOUN bracis-19024 243 13 of of ADP bracis-19024 243 14 the the DET bracis-19024 243 15 mdp mdp PROPN bracis-19024 243 16 . . PUNCT bracis-19024 244 1 leaving leave VERB bracis-19024 244 2 it it PRON bracis-19024 244 3 in in ADP bracis-19024 244 4 the the DET bracis-19024 244 5 definition definition NOUN bracis-19024 244 6 yields yield VERB bracis-19024 244 7 a a DET bracis-19024 244 8 more more ADV bracis-19024 244 9 general general ADJ bracis-19024 244 10 formulation formulation NOUN bracis-19024 244 11 , , PUNCT bracis-19024 244 12 since since SCONJ bracis-19024 244 13 it it PRON bracis-19024 244 14 allows allow VERB bracis-19024 244 15 one one PRON bracis-19024 244 16 to to PART bracis-19024 244 17 model model VERB bracis-19024 244 18 continuous continuous ADJ bracis-19024 244 19 ( ( PUNCT bracis-19024 244 20 non non ADJ bracis-19024 244 21 - - ADJ bracis-19024 244 22 ending ending ADJ bracis-19024 244 23 ) ) PUNCT bracis-19024 244 24 learning learn VERB bracis-19024 244 25 settings setting NOUN bracis-19024 244 26 . . PUNCT bracis-19024 245 1 3.the 3.the DET bracis-19024 245 2 value value NOUN bracis-19024 245 3 shown show VERB bracis-19024 245 4 for for ADP bracis-19024 245 5 \(r_2\ \(r_2\ VERB bracis-19024 245 6 ) ) PUNCT bracis-19024 245 7 might might AUX bracis-19024 245 8 contradict contradict VERB bracis-19024 245 9 the the DET bracis-19024 245 10 previous previous ADJ bracis-19024 245 11 discussion discussion NOUN bracis-19024 245 12 , , PUNCT bracis-19024 245 13 but but CCONJ bracis-19024 245 14 the the DET bracis-19024 245 15 mdp mdp NOUN bracis-19024 245 16 is be AUX bracis-19024 245 17 set set VERB bracis-19024 245 18 in in ADP bracis-19024 245 19 a a DET bracis-19024 245 20 way way NOUN bracis-19024 245 21 that that PRON bracis-19024 245 22 , , PUNCT bracis-19024 245 23 when when SCONJ bracis-19024 245 24 jobs job NOUN bracis-19024 245 25 are be AUX bracis-19024 245 26 scheduled schedule VERB bracis-19024 245 27 successfully successfully ADV bracis-19024 245 28 , , PUNCT bracis-19024 245 29 \(r_{t+1}=0\ \(r_{t+1}=0\ PROPN bracis-19024 245 30 ) ) PUNCT bracis-19024 245 31 . . PUNCT bracis-19024 246 1 4.in 4.in PRON bracis-19024 247 1 our our PRON bracis-19024 247 2 example example NOUN bracis-19024 247 3 , , PUNCT bracis-19024 247 4 for for ADP bracis-19024 247 5 each each DET bracis-19024 247 6 job job NOUN bracis-19024 247 7 \(j_i\ \(j_i\ NOUN bracis-19024 247 8 ) ) PUNCT bracis-19024 247 9 , , PUNCT bracis-19024 247 10 in in ADP bracis-19024 247 11 time time NOUN bracis-19024 247 12 step step NOUN bracis-19024 247 13 1 1 NUM bracis-19024 247 14 , , PUNCT bracis-19024 247 15 \(\pi \(\pi PROPN bracis-19024 247 16 \ \ NOUN bracis-19024 247 17 ) ) PUNCT bracis-19024 247 18 would would AUX bracis-19024 247 19 give give VERB bracis-19024 247 20 the the DET bracis-19024 247 21 probabilities probability NOUN bracis-19024 247 22 of of ADP bracis-19024 247 23 choosing choose VERB bracis-19024 247 24 each each DET bracis-19024 247 25 job job NOUN bracis-19024 247 26 given give VERB bracis-19024 247 27 an an DET bracis-19024 247 28 empty empty ADJ bracis-19024 247 29 cluster cluster NOUN bracis-19024 247 30 : : PUNCT bracis-19024 247 31 , , PUNCT bracis-19024 247 32 , , PUNCT bracis-19024 247 33 and and CCONJ bracis-19024 247 34 such such ADJ bracis-19024 247 35 that that SCONJ bracis-19024 247 36 , , PUNCT bracis-19024 247 37 by by ADP bracis-19024 247 38 total total ADJ bracis-19024 247 39 probability probability NOUN bracis-19024 247 40 , , PUNCT bracis-19024 247 41 . . PUNCT bracis-19024 248 1 5.normalization 5.normalization NUM bracis-19024 248 2 is be AUX bracis-19024 248 3 needed need VERB bracis-19024 248 4 to to PART bracis-19024 248 5 approximate approximate VERB bracis-19024 248 6 the the DET bracis-19024 248 7 average average ADJ bracis-19024 248 8 value value NOUN bracis-19024 248 9 of of ADP bracis-19024 248 10 \(\widehat{j(\theta \(\widehat{j(\theta PROPN bracis-19024 248 11 ) ) PUNCT bracis-19024 248 12 } } PUNCT bracis-19024 248 13 \ \ NOUN bracis-19024 248 14 ) ) PUNCT bracis-19024 248 15 . . PUNCT bracis-19024 249 1 otherwise otherwise ADV bracis-19024 249 2 , , PUNCT bracis-19024 249 3 \(\widehat{j(\theta \(\widehat{j(\theta PROPN bracis-19024 249 4 ) ) PUNCT bracis-19024 249 5 } } PUNCT bracis-19024 249 6 \rightarrow \rightarrow PROPN bracis-19024 249 7 \infty \infty PROPN bracis-19024 249 8 \ \ PROPN bracis-19024 249 9 ) ) PUNCT bracis-19024 249 10 as as ADP bracis-19024 249 11 \(n\rightarrow \(n\rightarrow NOUN bracis-19024 249 12 \infty \infty PROPN bracis-19024 249 13 \ \ PROPN bracis-19024 249 14 ) ) PUNCT bracis-19024 249 15 . . PUNCT bracis-19024 250 1 6.jobs 6.jobs NUM bracis-19024 250 2 in in ADP bracis-19024 250 3 the the DET bracis-19024 250 4 wait wait NOUN bracis-19024 250 5 queue queue NOUN bracis-19024 250 6 that that SCONJ bracis-19024 250 7 the the DET bracis-19024 250 8 agent agent NOUN bracis-19024 250 9 can can AUX bracis-19024 250 10 not not PART bracis-19024 250 11 choose choose VERB bracis-19024 250 12 to to PART bracis-19024 250 13 schedule schedule VERB bracis-19024 250 14 . . PUNCT bracis-19024 251 1 7.truncating 7.truncating NUM bracis-19024 251 2 the the DET bracis-19024 251 3 list list NOUN bracis-19024 251 4 of of ADP bracis-19024 251 5 jobs job NOUN bracis-19024 251 6 violates violate VERB bracis-19024 251 7 the the DET bracis-19024 251 8 markov markov NOUN bracis-19024 251 9 property property NOUN bracis-19024 251 10 , , PUNCT bracis-19024 251 11 since since SCONJ bracis-19024 251 12 once once SCONJ bracis-19024 251 13 it it PRON bracis-19024 251 14 overflows overflow VERB bracis-19024 251 15 , , PUNCT bracis-19024 251 16 the the DET bracis-19024 251 17 agent agent NOUN bracis-19024 251 18 can can AUX bracis-19024 251 19 not not PART bracis-19024 251 20 know know VERB bracis-19024 251 21 how how SCONJ bracis-19024 251 22 many many ADJ bracis-19024 251 23 jobs job NOUN bracis-19024 251 24 are be AUX bracis-19024 251 25 in in ADP bracis-19024 251 26 the the DET bracis-19024 251 27 system system NOUN bracis-19024 251 28 . . PUNCT bracis-19024 252 1 8.parentheses 8.parentheses NUM bracis-19024 252 2 group group NOUN bracis-19024 252 3 elements element NOUN bracis-19024 252 4 . . PUNCT bracis-19024 253 1 in in ADP bracis-19024 253 2 the the DET bracis-19024 253 3 first first ADJ bracis-19024 253 4 vector vector NOUN bracis-19024 253 5 , , PUNCT bracis-19024 253 6 there there PRON bracis-19024 253 7 are be VERB bracis-19024 253 8 five five NUM bracis-19024 253 9 parenthesized parenthesized ADJ bracis-19024 253 10 pairs pair NOUN bracis-19024 253 11 to to PART bracis-19024 253 12 indicate indicate VERB bracis-19024 253 13 the the DET bracis-19024 253 14 time time NOUN bracis-19024 253 15 horizon horizon NOUN bracis-19024 253 16 of of ADP bracis-19024 253 17 5 5 NUM bracis-19024 253 18 , , PUNCT bracis-19024 253 19 and and CCONJ bracis-19024 253 20 two two NUM bracis-19024 253 21 parenthesized parenthesized ADJ bracis-19024 253 22 elements element NOUN bracis-19024 253 23 to to PART bracis-19024 253 24 represent represent VERB bracis-19024 253 25 job job NOUN bracis-19024 253 26 slows slow NOUN bracis-19024 253 27 in in ADP bracis-19024 253 28 window window NOUN bracis-19024 253 29 w. w. PROPN bracis-19024 253 30 references reference NOUN bracis-19024 253 31 cunha cunha PROPN bracis-19024 253 32 , , PUNCT bracis-19024 253 33 r.l r.l PROPN bracis-19024 253 34 . . PROPN bracis-19024 253 35 , , PUNCT bracis-19024 253 36 rodrigues rodrigues PROPN bracis-19024 253 37 , , PUNCT bracis-19024 253 38 e.r e.r PROPN bracis-19024 253 39 . . PROPN bracis-19024 253 40 , , PUNCT bracis-19024 253 41 tizzei tizzei PROPN bracis-19024 253 42 , , PUNCT bracis-19024 253 43 l.p l.p PROPN bracis-19024 253 44 . . PROPN bracis-19024 253 45 , , PUNCT bracis-19024 253 46 netto netto PROPN bracis-19024 253 47 , , PUNCT bracis-19024 253 48 m.a.s m.a.s PROPN bracis-19024 253 49 . . PUNCT bracis-19024 253 50 : : PUNCT bracis-19024 254 1 job job NOUN bracis-19024 254 2 placement placement NOUN bracis-19024 254 3 advisor advisor NOUN bracis-19024 254 4 based base VERB bracis-19024 254 5 on on ADP bracis-19024 254 6 turnaround turnaround NOUN bracis-19024 254 7 predictions prediction NOUN bracis-19024 254 8 for for ADP bracis-19024 254 9 hpc hpc NOUN bracis-19024 254 10 hybrid hybrid ADJ bracis-19024 254 11 clouds cloud NOUN bracis-19024 254 12 . . PUNCT bracis-19024 255 1 future future ADJ bracis-19024 255 2 gener gener PROPN bracis-19024 255 3 . . PUNCT bracis-19024 256 1 comput comput PROPN bracis-19024 256 2 . . PUNCT bracis-19024 257 1 syst syst PROPN bracis-19024 257 2 . . PROPN bracis-19024 258 1 67 67 NUM bracis-19024 258 2 , , PUNCT bracis-19024 258 3 35–46 35–46 NUM bracis-19024 258 4 ( ( PUNCT bracis-19024 258 5 2017 2017 NUM bracis-19024 258 6 ) ) PUNCT bracis-19024 258 7 . . PUNCT bracis-19024 259 1 issn issn PROPN bracis-19024 259 2 0167–739x 0167–739x PROPN bracis-19024 259 3 article article PROPN bracis-19024 259 4     SPACE bracis-19024 259 5 google google PROPN bracis-19024 259 6 scholar scholar NOUN bracis-19024 259 7     SPACE bracis-19024 259 8 de de X bracis-19024 259 9 freitas freitas PROPN bracis-19024 259 10 cunha cunha NOUN bracis-19024 259 11 , , PUNCT bracis-19024 259 12 r.l r.l PROPN bracis-19024 259 13 . . PROPN bracis-19024 259 14 , , PUNCT bracis-19024 259 15 chaimowicz chaimowicz PROPN bracis-19024 259 16 , , PUNCT bracis-19024 259 17 l. l. PROPN bracis-19024 259 18 : : PUNCT bracis-19024 259 19 towards towards ADP bracis-19024 259 20 a a DET bracis-19024 259 21 common common ADJ bracis-19024 259 22 environment environment NOUN bracis-19024 259 23 for for ADP bracis-19024 259 24 learning learn VERB bracis-19024 259 25 scheduling scheduling NOUN bracis-19024 259 26 algorithms algorithm NOUN bracis-19024 259 27 . . PUNCT bracis-19024 260 1 in in ADP bracis-19024 260 2 : : PUNCT bracis-19024 260 3 2020 2020 NUM bracis-19024 260 4 28th 28th ADJ bracis-19024 260 5 international international ADJ bracis-19024 260 6 symposium symposium NOUN bracis-19024 260 7 on on ADP bracis-19024 260 8 modeling modeling NOUN bracis-19024 260 9 , , PUNCT bracis-19024 260 10 analysis analysis NOUN bracis-19024 260 11 , , PUNCT bracis-19024 260 12 and and CCONJ bracis-19024 260 13 simulation simulation NOUN bracis-19024 260 14 of of ADP bracis-19024 260 15 computer computer NOUN bracis-19024 260 16 and and CCONJ bracis-19024 260 17 telecommunication telecommunication NOUN bracis-19024 260 18 systems system NOUN bracis-19024 260 19 ( ( PUNCT bracis-19024 260 20 mascots mascot NOUN bracis-19024 260 21 ) ) PUNCT bracis-19024 260 22 , , PUNCT bracis-19024 260 23 pp pp ADP bracis-19024 260 24 . . PUNCT bracis-19024 261 1 1–8 1–8 NUM bracis-19024 261 2 ( ( PUNCT bracis-19024 261 3 2020 2020 NUM bracis-19024 261 4 ) ) PUNCT bracis-19024 261 5 google google PROPN bracis-19024 261 6 scholar scholar NOUN bracis-19024 261 7     SPACE bracis-19024 261 8 domeniconi domeniconi NOUN bracis-19024 261 9 , , PUNCT bracis-19024 261 10 g. g. PROPN bracis-19024 261 11 , , PUNCT bracis-19024 261 12 lee lee PROPN bracis-19024 261 13 , , PUNCT bracis-19024 261 14 e.k e.k PROPN bracis-19024 261 15 . . PROPN bracis-19024 261 16 , , PUNCT bracis-19024 261 17 morari morari PROPN bracis-19024 261 18 , , PUNCT bracis-19024 261 19 a. a. NOUN bracis-19024 261 20 : : PUNCT bracis-19024 261 21 cush cush NOUN bracis-19024 261 22 : : PUNCT bracis-19024 261 23 cognitive cognitive ADJ bracis-19024 261 24 scheduler scheduler NOUN bracis-19024 261 25 for for ADP bracis-19024 261 26 heterogeneous heterogeneous ADJ bracis-19024 261 27 high high ADJ bracis-19024 261 28 performance performance NOUN bracis-19024 261 29 computing computing NOUN bracis-19024 261 30 system system NOUN bracis-19024 261 31 . . PUNCT bracis-19024 262 1 in in ADP bracis-19024 262 2 : : PUNCT bracis-19024 262 3 proceedings proceeding NOUN bracis-19024 262 4 of of ADP bracis-19024 262 5 drl4kdd drl4kdd ADJ bracis-19024 262 6 19 19 NUM bracis-19024 262 7 : : PUNCT bracis-19024 262 8 workshop workshop NOUN bracis-19024 262 9 on on ADP bracis-19024 262 10 deep deep ADJ bracis-19024 262 11 reinforcement reinforcement NOUN bracis-19024 262 12 learning learning NOUN bracis-19024 262 13 for for ADP bracis-19024 262 14 knowledge knowledge NOUN bracis-19024 262 15 discovery discovery NOUN bracis-19024 262 16 ( ( PUNCT bracis-19024 262 17 drl4kdd drl4kdd NOUN bracis-19024 262 18 ) ) PUNCT bracis-19024 262 19 , , PUNCT bracis-19024 262 20 vol vol NOUN bracis-19024 262 21 . . PROPN bracis-19024 262 22 12 12 NUM bracis-19024 262 23 ( ( PUNCT bracis-19024 262 24 2019 2019 NUM bracis-19024 262 25 ) ) PUNCT bracis-19024 262 26 google google PROPN bracis-19024 262 27 scholar scholar NOUN bracis-19024 262 28     SPACE bracis-19024 262 29 fan fan NOUN bracis-19024 262 30 , , PUNCT bracis-19024 262 31 y. y. PROPN bracis-19024 262 32 , , PUNCT bracis-19024 262 33 lan lan PROPN bracis-19024 262 34 , , PUNCT bracis-19024 262 35 z. z. PROPN bracis-19024 262 36 , , PUNCT bracis-19024 262 37 childers childers PROPN bracis-19024 262 38 , , PUNCT bracis-19024 262 39 t. t. PROPN bracis-19024 262 40 , , PUNCT bracis-19024 262 41 rich rich ADJ bracis-19024 262 42 , , PUNCT bracis-19024 262 43 p. p. NOUN bracis-19024 262 44 , , PUNCT bracis-19024 262 45 allcock allcock PROPN bracis-19024 262 46 , , PUNCT bracis-19024 262 47 w. w. PROPN bracis-19024 262 48 , , PUNCT bracis-19024 262 49 papka papka PROPN bracis-19024 262 50 , , PUNCT bracis-19024 262 51 m.e m.e PROPN bracis-19024 262 52 . . PROPN bracis-19024 262 53 : : PUNCT bracis-19024 263 1 deep deep ADJ bracis-19024 263 2 reinforcement reinforcement NOUN bracis-19024 263 3 agent agent NOUN bracis-19024 263 4 for for ADP bracis-19024 263 5 scheduling scheduling NOUN bracis-19024 263 6 in in ADP bracis-19024 263 7 hpc hpc PROPN bracis-19024 263 8 . . PUNCT bracis-19024 264 1 arxiv arxiv PROPN bracis-19024 264 2 preprint preprint PROPN bracis-19024 264 3 arxiv:2102.06243 arxiv:2102.06243 NOUN bracis-19024 264 4 ( ( PUNCT bracis-19024 264 5 2021 2021 NUM bracis-19024 264 6 ) ) PUNCT bracis-19024 264 7 feitelson feitelson NOUN bracis-19024 264 8 , , PUNCT bracis-19024 264 9 d.g d.g PROPN bracis-19024 264 10 . . PROPN bracis-19024 264 11 , , PUNCT bracis-19024 264 12 rudolph rudolph PROPN bracis-19024 264 13 , , PUNCT bracis-19024 264 14 l. l. PROPN bracis-19024 264 15 : : PUNCT bracis-19024 264 16 toward toward ADP bracis-19024 264 17 convergence convergence NOUN bracis-19024 264 18 in in ADP bracis-19024 264 19 job job NOUN bracis-19024 264 20 schedulers scheduler NOUN bracis-19024 264 21 for for ADP bracis-19024 264 22 parallel parallel ADJ bracis-19024 264 23 supercomputers supercomputer NOUN bracis-19024 264 24 . . PUNCT bracis-19024 265 1 in in ADP bracis-19024 265 2 : : PUNCT bracis-19024 265 3 feitelson feitelson NOUN bracis-19024 265 4 , , PUNCT bracis-19024 265 5 d.g d.g PROPN bracis-19024 265 6 . . PROPN bracis-19024 265 7 , , PUNCT bracis-19024 265 8 rudolph rudolph PROPN bracis-19024 265 9 , , PUNCT bracis-19024 265 10 l. l. PROPN bracis-19024 265 11 ( ( PUNCT bracis-19024 265 12 eds eds PROPN bracis-19024 265 13 . . PUNCT bracis-19024 265 14 ) ) PUNCT bracis-19024 266 1 jsspp jsspp NOUN bracis-19024 266 2 1996 1996 NUM bracis-19024 266 3 . . PUNCT bracis-19024 267 1 lncs lncs PROPN bracis-19024 267 2 , , PUNCT bracis-19024 267 3 vol vol NOUN bracis-19024 267 4 . . PROPN bracis-19024 267 5 1162 1162 NUM bracis-19024 267 6 , , PUNCT bracis-19024 267 7 pp pp ADP bracis-19024 267 8 . . PUNCT bracis-19024 268 1 1–26 1–26 NOUN bracis-19024 268 2 . . PUNCT bracis-19024 269 1 springer springer NOUN bracis-19024 269 2 , , PUNCT bracis-19024 269 3 heidelberg heidelberg PROPN bracis-19024 269 4 ( ( PUNCT bracis-19024 269 5 1996 1996 NUM bracis-19024 269 6 ) ) PUNCT bracis-19024 269 7 . . PUNCT bracis-19024 270 1 https://doi.org/10.1007/bfb0022284 https://doi.org/10.1007/bfb0022284 NOUN bracis-19024 270 2 chapter chapter NOUN bracis-19024 270 3     SPACE bracis-19024 270 4 math math NOUN bracis-19024 270 5     SPACE bracis-19024 270 6 google google PROPN bracis-19024 270 7 scholar scholar NOUN bracis-19024 270 8     SPACE bracis-19024 270 9 liang liang PROPN bracis-19024 270 10 , , PUNCT bracis-19024 270 11 y. y. PROPN bracis-19024 270 12 , , PUNCT bracis-19024 270 13 machado machado PROPN bracis-19024 270 14 , , PUNCT bracis-19024 270 15 m.c m.c PROPN bracis-19024 270 16 . . PROPN bracis-19024 270 17 , , PUNCT bracis-19024 270 18 talvitie talvitie INTJ bracis-19024 270 19 , , PUNCT bracis-19024 270 20 e. e. PROPN bracis-19024 270 21 , , PUNCT bracis-19024 270 22 bowling bowling NOUN bracis-19024 270 23 , , PUNCT bracis-19024 270 24 m. m. NOUN bracis-19024 270 25 : : PUNCT bracis-19024 270 26 state state NOUN bracis-19024 270 27 of of ADP bracis-19024 270 28 the the DET bracis-19024 270 29 art art NOUN bracis-19024 270 30 control control NOUN bracis-19024 270 31 of of ADP bracis-19024 270 32 atari atari PROPN bracis-19024 270 33 games game NOUN bracis-19024 270 34 using use VERB bracis-19024 270 35 shallow shallow ADJ bracis-19024 270 36 reinforcement reinforcement NOUN bracis-19024 270 37 learning learning NOUN bracis-19024 270 38 . . PUNCT bracis-19024 271 1 in in ADP bracis-19024 271 2 aamas aamas PROPN bracis-19024 271 3 ( ( PUNCT bracis-19024 271 4 2016 2016 NUM bracis-19024 271 5 ) ) PUNCT bracis-19024 271 6 google google PROPN bracis-19024 271 7 scholar scholar NOUN bracis-19024 271 8     SPACE bracis-19024 271 9 mao mao PROPN bracis-19024 271 10 , , PUNCT bracis-19024 271 11 h. h. PROPN bracis-19024 271 12 , , PUNCT bracis-19024 271 13 alizadeh alizadeh NOUN bracis-19024 271 14 , , PUNCT bracis-19024 271 15 m. m. NOUN bracis-19024 271 16 , , PUNCT bracis-19024 271 17 menache menache PROPN bracis-19024 271 18 , , PUNCT bracis-19024 271 19 i. i. PROPN bracis-19024 271 20 , , PUNCT bracis-19024 271 21 kandula kandula PROPN bracis-19024 271 22 , , PUNCT bracis-19024 271 23 s. s. PROPN bracis-19024 271 24 : : PUNCT bracis-19024 271 25 resource resource NOUN bracis-19024 271 26 management management NOUN bracis-19024 271 27 with with ADP bracis-19024 271 28 deep deep ADJ bracis-19024 271 29 reinforcement reinforcement NOUN bracis-19024 271 30 learning learning NOUN bracis-19024 271 31 . . PUNCT bracis-19024 272 1 in in ADP bracis-19024 272 2 : : PUNCT bracis-19024 272 3 proceedings proceeding NOUN bracis-19024 272 4 of of ADP bracis-19024 272 5 the the DET bracis-19024 272 6 15th 15th ADJ bracis-19024 272 7 acm acm NOUN bracis-19024 272 8 workshop workshop NOUN bracis-19024 272 9 on on ADP bracis-19024 272 10 hot hot ADJ bracis-19024 272 11 topics topic NOUN bracis-19024 272 12 in in ADP bracis-19024 272 13 networks network NOUN bracis-19024 272 14 , , PUNCT bracis-19024 272 15 pp pp ADV bracis-19024 272 16 . . PUNCT bracis-19024 273 1 50–56 50–56 NUM bracis-19024 273 2 ( ( PUNCT bracis-19024 273 3 2016 2016 NUM bracis-19024 273 4 ) ) PUNCT bracis-19024 273 5 google google PROPN bracis-19024 273 6 scholar scholar NOUN bracis-19024 273 7     SPACE bracis-19024 273 8 mnih mnih PROPN bracis-19024 273 9 , , PUNCT bracis-19024 273 10 v. v. PROPN bracis-19024 273 11 , , PUNCT bracis-19024 273 12 et et PROPN bracis-19024 273 13 al al PROPN bracis-19024 273 14 . . PUNCT bracis-19024 273 15 : : PUNCT bracis-19024 273 16 asynchronous asynchronous ADJ bracis-19024 273 17 methods method NOUN bracis-19024 273 18 for for ADP bracis-19024 273 19 deep deep ADJ bracis-19024 273 20 reinforcement reinforcement NOUN bracis-19024 273 21 learning learning NOUN bracis-19024 273 22 . . PUNCT bracis-19024 274 1 in in ADP bracis-19024 274 2 : : PUNCT bracis-19024 274 3 international international ADJ bracis-19024 274 4 conference conference NOUN bracis-19024 274 5 on on ADP bracis-19024 274 6 machine machine NOUN bracis-19024 274 7 learning learning NOUN bracis-19024 274 8 , , PUNCT bracis-19024 274 9 pp pp ADP bracis-19024 274 10 . . PUNCT bracis-19024 274 11 1928–1937 1928–1937 NUM bracis-19024 274 12 ( ( PUNCT bracis-19024 274 13 2016 2016 NUM bracis-19024 274 14 ) ) PUNCT bracis-19024 274 15 google google PROPN bracis-19024 274 16 scholar scholar NOUN bracis-19024 274 17     SPACE bracis-19024 274 18 nishida nishida PROPN bracis-19024 274 19 , , PUNCT bracis-19024 274 20 c.e.h c.e.h PROPN bracis-19024 274 21 . . PROPN bracis-19024 274 22 , , PUNCT bracis-19024 274 23 costa costa PROPN bracis-19024 274 24 , , PUNCT bracis-19024 274 25 a.h.r a.h.r PROPN bracis-19024 274 26 . . PROPN bracis-19024 274 27 , , PUNCT bracis-19024 274 28 da da PROPN bracis-19024 274 29 costa costa PROPN bracis-19024 274 30 bianchi bianchi PROPN bracis-19024 274 31 , , PUNCT bracis-19024 274 32 r.a r.a PROPN bracis-19024 274 33 . . PROPN bracis-19024 274 34 : : PUNCT bracis-19024 275 1 control control NOUN bracis-19024 275 2 of of ADP bracis-19024 275 3 gene gene NOUN bracis-19024 275 4 regulatory regulatory ADJ bracis-19024 275 5 networks network NOUN bracis-19024 275 6 basin basin NOUN bracis-19024 275 7 of of ADP bracis-19024 275 8 attractions attraction NOUN bracis-19024 275 9 with with ADP bracis-19024 275 10 batch batch NOUN bracis-19024 275 11 reinforcement reinforcement NOUN bracis-19024 275 12 learning learning NOUN bracis-19024 275 13 . . PUNCT bracis-19024 276 1 in in ADP bracis-19024 276 2 : : PUNCT bracis-19024 276 3 2018 2018 NUM bracis-19024 276 4 7th 7th ADJ bracis-19024 276 5 brazilian brazilian ADJ bracis-19024 276 6 conference conference NOUN bracis-19024 276 7 on on ADP bracis-19024 276 8 intelligent intelligent ADJ bracis-19024 276 9 systems system NOUN bracis-19024 276 10 ( ( PUNCT bracis-19024 276 11 bracis bracis NOUN bracis-19024 276 12 ) ) PUNCT bracis-19024 276 13 , , PUNCT bracis-19024 276 14 pp pp ADP bracis-19024 276 15 . . PUNCT bracis-19024 277 1 127–132 127–132 NUM bracis-19024 277 2 ( ( PUNCT bracis-19024 277 3 2018 2018 NUM bracis-19024 277 4 ) ) PUNCT bracis-19024 277 5 google google NOUN bracis-19024 277 6 scholar scholar NOUN bracis-19024 277 7     SPACE bracis-19024 277 8 raffin raffin NOUN bracis-19024 277 9 , , PUNCT bracis-19024 277 10 a. a. PROPN bracis-19024 277 11 , , PUNCT bracis-19024 277 12 hill hill PROPN bracis-19024 277 13 , , PUNCT bracis-19024 277 14 a. a. NOUN bracis-19024 277 15 , , PUNCT bracis-19024 277 16 ernestus ernestus NOUN bracis-19024 277 17 , , PUNCT bracis-19024 277 18 m. m. NOUN bracis-19024 277 19 , , PUNCT bracis-19024 277 20 gleave gleave PROPN bracis-19024 277 21 , , PUNCT bracis-19024 277 22 a. a. NOUN bracis-19024 277 23 , , PUNCT bracis-19024 277 24 kanervisto kanervisto PROPN bracis-19024 277 25 , , PUNCT bracis-19024 277 26 a. a. NOUN bracis-19024 277 27 , , PUNCT bracis-19024 277 28 dormann dormann NOUN bracis-19024 277 29 , , PUNCT bracis-19024 277 30 n. n. NOUN bracis-19024 277 31 : : PUNCT bracis-19024 277 32 stable stable ADJ bracis-19024 277 33 baselines3 baselines3 NOUN bracis-19024 277 34 ( ( PUNCT bracis-19024 277 35 2019 2019 NUM bracis-19024 277 36 ) ) PUNCT bracis-19024 277 37 . . PUNCT bracis-19024 278 1 https://github.com/dlr-rm/stable-baselines3 https://github.com/dlr-rm/stable-baselines3 PROPN bracis-19024 278 2 ramos ramos PROPN bracis-19024 278 3 , , PUNCT bracis-19024 278 4 w. w. PROPN bracis-19024 278 5 , , PUNCT bracis-19024 278 6 silva silva PROPN bracis-19024 278 7 , , PUNCT bracis-19024 278 8 m. m. NOUN bracis-19024 278 9 , , PUNCT bracis-19024 278 10 araujo araujo PROPN bracis-19024 278 11 , , PUNCT bracis-19024 278 12 e. e. PROPN bracis-19024 278 13 , , PUNCT bracis-19024 278 14 marcolino marcolino PROPN bracis-19024 278 15 , , PUNCT bracis-19024 278 16 l.s l.s PROPN bracis-19024 278 17 . . PROPN bracis-19024 278 18 , , PUNCT bracis-19024 278 19 nascimento nascimento PROPN bracis-19024 278 20 , , PUNCT bracis-19024 278 21 e. e. PROPN bracis-19024 278 22 : : PUNCT bracis-19024 278 23 straight straight ADV bracis-19024 278 24 to to ADP bracis-19024 278 25 the the DET bracis-19024 278 26 point point NOUN bracis-19024 278 27 : : PUNCT bracis-19024 278 28 fast fast ADJ bracis-19024 278 29 - - PUNCT bracis-19024 278 30 forwarding forward VERB bracis-19024 278 31 videos video NOUN bracis-19024 278 32 via via ADP bracis-19024 278 33 reinforcement reinforcement NOUN bracis-19024 278 34 learning learning NOUN bracis-19024 278 35 using use VERB bracis-19024 278 36 textual textual ADJ bracis-19024 278 37 data datum NOUN bracis-19024 278 38 . . PUNCT bracis-19024 279 1 in in ADP bracis-19024 279 2 : : PUNCT bracis-19024 279 3 proceedings proceeding NOUN bracis-19024 279 4 of of ADP bracis-19024 279 5 the the DET bracis-19024 279 6 ieee ieee NOUN bracis-19024 279 7 / / SYM bracis-19024 279 8 cvf cvf NOUN bracis-19024 279 9 conference conference NOUN bracis-19024 279 10 on on ADP bracis-19024 279 11 computer computer NOUN bracis-19024 279 12 vision vision NOUN bracis-19024 279 13 and and CCONJ bracis-19024 279 14 pattern pattern NOUN bracis-19024 279 15 recognition recognition NOUN bracis-19024 279 16 , , PUNCT bracis-19024 279 17 pp pp ADP bracis-19024 279 18 . . PUNCT bracis-19024 280 1 10931–10940 10931–10940 NUM bracis-19024 280 2 ( ( PUNCT bracis-19024 280 3 2020 2020 NUM bracis-19024 280 4 ) ) PUNCT bracis-19024 280 5 google google PROPN bracis-19024 280 6 scholar scholar NOUN bracis-19024 280 7     SPACE bracis-19024 280 8 schulman schulman PROPN bracis-19024 280 9 , , PUNCT bracis-19024 280 10 j. j. PROPN bracis-19024 280 11 , , PUNCT bracis-19024 280 12 wolski wolski PROPN bracis-19024 280 13 , , PUNCT bracis-19024 280 14 f. f. PROPN bracis-19024 280 15 , , PUNCT bracis-19024 280 16 dhariwal dhariwal NOUN bracis-19024 280 17 , , PUNCT bracis-19024 280 18 p. p. PROPN bracis-19024 280 19 , , PUNCT bracis-19024 280 20 radford radford PROPN bracis-19024 280 21 , , PUNCT bracis-19024 280 22 a. a. PROPN bracis-19024 280 23 , , PUNCT bracis-19024 280 24 klimov klimov PROPN bracis-19024 280 25 , , PUNCT bracis-19024 280 26 o. o. NOUN bracis-19024 280 27 : : PUNCT bracis-19024 280 28 proximal proximal ADJ bracis-19024 280 29 policy policy NOUN bracis-19024 280 30 optimization optimization NOUN bracis-19024 280 31 algorithms algorithm NOUN bracis-19024 280 32 . . PUNCT bracis-19024 281 1 arxiv arxiv PROPN bracis-19024 281 2 preprint preprint VERB bracis-19024 281 3 arxiv:1707.06347 arxiv:1707.06347 ADJ bracis-19024 281 4 ( ( PUNCT bracis-19024 281 5 2017 2017 NUM bracis-19024 281 6 ) ) PUNCT bracis-19024 281 7 silver silver NOUN bracis-19024 281 8 , , PUNCT bracis-19024 281 9 d. d. PROPN bracis-19024 281 10 , , PUNCT bracis-19024 281 11 et et PROPN bracis-19024 281 12 al al PROPN bracis-19024 281 13 . . PROPN bracis-19024 281 14 : : PUNCT bracis-19024 282 1 a a DET bracis-19024 282 2 general general ADJ bracis-19024 282 3 reinforcement reinforcement NOUN bracis-19024 282 4 learning learning NOUN bracis-19024 282 5 algorithm algorithm NOUN bracis-19024 282 6 that that SCONJ bracis-19024 282 7 masters master NOUN bracis-19024 282 8 chess chess NOUN bracis-19024 282 9 , , PUNCT bracis-19024 282 10 shogi shogi NOUN bracis-19024 282 11 , , PUNCT bracis-19024 282 12 and and CCONJ bracis-19024 282 13 go go VERB bracis-19024 282 14 through through ADP bracis-19024 282 15 self self NOUN bracis-19024 282 16 - - PUNCT bracis-19024 282 17 play play NOUN bracis-19024 282 18 . . PUNCT bracis-19024 283 1 science science NOUN bracis-19024 283 2 362(6419 362(6419 NUM bracis-19024 283 3 ) ) PUNCT bracis-19024 283 4 , , PUNCT bracis-19024 283 5 1140–1144 1140–1144 NUM bracis-19024 283 6 ( ( PUNCT bracis-19024 283 7 2018 2018 NUM bracis-19024 283 8 ) ) PUNCT bracis-19024 283 9 article article NOUN bracis-19024 283 10     SPACE bracis-19024 283 11 mathscinet mathscinet NOUN bracis-19024 283 12     SPACE bracis-19024 283 13 google google PROPN bracis-19024 283 14 scholar scholar NOUN bracis-19024 283 15     SPACE bracis-19024 283 16 sutton sutton PROPN bracis-19024 283 17 , , PUNCT bracis-19024 283 18 r.s r.s PROPN bracis-19024 283 19 . . PROPN bracis-19024 283 20 , , PUNCT bracis-19024 283 21 precup precup PROPN bracis-19024 283 22 , , PUNCT bracis-19024 283 23 d. d. PROPN bracis-19024 283 24 , , PUNCT bracis-19024 283 25 singh singh PROPN bracis-19024 283 26 , , PUNCT bracis-19024 283 27 s. s. PROPN bracis-19024 283 28 : : PUNCT bracis-19024 283 29 between between ADP bracis-19024 283 30 mdps mdps NOUN bracis-19024 283 31 and and CCONJ bracis-19024 283 32 semi semi ADJ bracis-19024 283 33 - - ADJ bracis-19024 283 34 mdps mdps ADJ bracis-19024 283 35 : : PUNCT bracis-19024 283 36 a a DET bracis-19024 283 37 framework framework NOUN bracis-19024 283 38 for for ADP bracis-19024 283 39 temporal temporal ADJ bracis-19024 283 40 abstraction abstraction NOUN bracis-19024 283 41 in in ADP bracis-19024 283 42 reinforcement reinforcement NOUN bracis-19024 283 43 learning learning NOUN bracis-19024 283 44 . . PUNCT bracis-19024 284 1 artif artif INTJ bracis-19024 284 2 . . PUNCT bracis-19024 285 1 intell intell PROPN bracis-19024 285 2 . . PUNCT bracis-19024 286 1 112(1 112(1 NUM bracis-19024 286 2 ) ) PUNCT bracis-19024 286 3 , , PUNCT bracis-19024 286 4 181–211 181–211 NUM bracis-19024 286 5 ( ( PUNCT bracis-19024 286 6 1999 1999 NUM bracis-19024 286 7 ) ) PUNCT bracis-19024 286 8 . . PUNCT bracis-19024 287 1 https://doi.org/10.1016/s0004-3702(99)00052-1 https://doi.org/10.1016/s0004-3702(99)00052-1 PROPN bracis-19024 287 2 . . PUNCT bracis-19024 288 1 issn issn PROPN bracis-19024 288 2 0004–3702 0004–3702 NUM bracis-19024 288 3 article article NOUN bracis-19024 288 4     SPACE bracis-19024 288 5 mathscinet mathscinet NOUN bracis-19024 288 6     SPACE bracis-19024 288 7 math math NOUN bracis-19024 288 8     SPACE bracis-19024 288 9 google google PROPN bracis-19024 288 10 scholar scholar NOUN bracis-19024 288 11     SPACE bracis-19024 288 12 tesauro tesauro NOUN bracis-19024 288 13 , , PUNCT bracis-19024 288 14 g. g. NOUN bracis-19024 288 15 : : PUNCT bracis-19024 288 16 td td NOUN bracis-19024 288 17 - - PUNCT bracis-19024 288 18 gammon gammon NOUN bracis-19024 288 19 , , PUNCT bracis-19024 288 20 a a DET bracis-19024 288 21 self self NOUN bracis-19024 288 22 - - PUNCT bracis-19024 288 23 teaching teach VERB bracis-19024 288 24 backgammon backgammon NOUN bracis-19024 288 25 program program NOUN bracis-19024 288 26 , , PUNCT bracis-19024 288 27 achieves achieve VERB bracis-19024 288 28 master master NOUN bracis-19024 288 29 - - PUNCT bracis-19024 288 30 level level NOUN bracis-19024 288 31 play play NOUN bracis-19024 288 32 . . PUNCT bracis-19024 289 1 neural neural ADJ bracis-19024 289 2 comput comput NOUN bracis-19024 289 3 . . PUNCT bracis-19024 290 1 6(2 6(2 NUM bracis-19024 290 2 ) ) PUNCT bracis-19024 290 3 , , PUNCT bracis-19024 290 4 215–219 215–219 NUM bracis-19024 290 5 ( ( PUNCT bracis-19024 290 6 1994 1994 NUM bracis-19024 290 7 ) ) PUNCT bracis-19024 290 8 article article NOUN bracis-19024 290 9     SPACE bracis-19024 290 10 google google PROPN bracis-19024 290 11 scholar scholar PROPN bracis-19024 290 12     SPACE bracis-19024 290 13 zhang zhang PROPN bracis-19024 290 14 , , PUNCT bracis-19024 290 15 d. d. PROPN bracis-19024 290 16 , , PUNCT bracis-19024 290 17 dai dai PROPN bracis-19024 290 18 , , PUNCT bracis-19024 290 19 d. d. PROPN bracis-19024 290 20 , , PUNCT bracis-19024 290 21 he he PRON bracis-19024 290 22 , , PUNCT bracis-19024 290 23 y. y. PROPN bracis-19024 290 24 , , PUNCT bracis-19024 290 25 bao bao PROPN bracis-19024 290 26 , , PUNCT bracis-19024 290 27 f.s f.s PROPN bracis-19024 290 28 . . PROPN bracis-19024 290 29 , , PUNCT bracis-19024 290 30 xie xie PROPN bracis-19024 290 31 , , PUNCT bracis-19024 290 32 b. b. PROPN bracis-19024 290 33 : : PUNCT bracis-19024 291 1 rlscheduler rlscheduler NOUN bracis-19024 291 2 : : PUNCT bracis-19024 291 3 an an DET bracis-19024 291 4 automated automate VERB bracis-19024 291 5 hpc hpc NOUN bracis-19024 291 6 batch batch NOUN bracis-19024 291 7 job job NOUN bracis-19024 291 8 scheduler scheduler NOUN bracis-19024 291 9 using use VERB bracis-19024 291 10 reinforcement reinforcement NOUN bracis-19024 291 11 learning learning NOUN bracis-19024 291 12 . . PUNCT bracis-19024 292 1 in in ADP bracis-19024 292 2 : : PUNCT bracis-19024 292 3 sc20 sc20 ADJ bracis-19024 292 4 : : PUNCT bracis-19024 292 5 international international ADJ bracis-19024 292 6 conference conference NOUN bracis-19024 292 7 for for ADP bracis-19024 292 8 high high ADJ bracis-19024 292 9 performance performance NOUN bracis-19024 292 10 computing computing NOUN bracis-19024 292 11 , , PUNCT bracis-19024 292 12 networking networking NOUN bracis-19024 292 13 , , PUNCT bracis-19024 292 14 storage storage NOUN bracis-19024 292 15 and and CCONJ bracis-19024 292 16 analysis analysis NOUN bracis-19024 292 17 , , PUNCT bracis-19024 292 18 pp pp ADJ bracis-19024 292 19 . . PUNCT bracis-19024 293 1 1–15 1–15 NOUN bracis-19024 293 2 . . PUNCT bracis-19024 294 1 ieee ieee NOUN bracis-19024 294 2 ( ( PUNCT bracis-19024 294 3 2020 2020 NUM bracis-19024 294 4 ) ) PUNCT bracis-19024 294 5 google google PROPN bracis-19024 294 6 scholar scholar NOUN bracis-19024 294 7     SPACE bracis-19024 294 8 download download NOUN bracis-19024 294 9 references reference NOUN bracis-19024 294 10 author author NOUN bracis-19024 294 11 information information NOUN bracis-19024 294 12 authors author NOUN bracis-19024 294 13 and and CCONJ bracis-19024 294 14 affiliations affiliation NOUN bracis-19024 294 15 programa programa PROPN bracis-19024 294 16 de de PROPN bracis-19024 294 17 pós pós PROPN bracis-19024 294 18 graduação graduação PROPN bracis-19024 294 19 em em PRON bracis-19024 294 20 ciência ciência PROPN bracis-19024 294 21 da da PROPN bracis-19024 294 22 computação computação PROPN bracis-19024 294 23 , , PUNCT bracis-19024 294 24 universidade universidade PROPN bracis-19024 294 25 federal federal PROPN bracis-19024 294 26 de de PROPN bracis-19024 294 27 minas minas PROPN bracis-19024 294 28 gerais gerais PROPN bracis-19024 294 29 ( ( PUNCT bracis-19024 294 30 ppgcc ppgcc NOUN bracis-19024 294 31 - - PUNCT bracis-19024 294 32 ufmg ufmg ADJ bracis-19024 294 33 ) ) PUNCT bracis-19024 294 34 , , PUNCT bracis-19024 294 35 belo belo PROPN bracis-19024 294 36 horizonte horizonte PROPN bracis-19024 294 37 , , PUNCT bracis-19024 294 38 mg mg PROPN bracis-19024 294 39 , , PUNCT bracis-19024 294 40 brazil brazil PROPN bracis-19024 294 41 renato renato PROPN bracis-19024 294 42 luiz luiz PROPN bracis-19024 294 43 de de PROPN bracis-19024 294 44 freitas freitas PROPN bracis-19024 294 45 cunha cunha NOUN bracis-19024 294 46     SPACE bracis-19024 294 47 & & CCONJ bracis-19024 294 48     SPACE bracis-19024 294 49 luiz luiz PROPN bracis-19024 294 50 chaimowicz chaimowicz PROPN bracis-19024 294 51 authors author NOUN bracis-19024 294 52 renato renato PROPN bracis-19024 294 53 luiz luiz PROPN bracis-19024 294 54 de de PROPN bracis-19024 294 55 freitas freitas PROPN bracis-19024 294 56 cunhaview cunhaview PROPN bracis-19024 294 57 author author NOUN bracis-19024 294 58 publications publication VERB bracis-19024 294 59 search search NOUN bracis-19024 294 60 author author NOUN bracis-19024 294 61 on on ADP bracis-19024 294 62 : : PUNCT bracis-19024 294 63 pubmed pubmed PROPN bracis-19024 294 64     SPACE bracis-19024 294 65 google google PROPN bracis-19024 294 66 scholar scholar NOUN bracis-19024 294 67 luiz luiz PROPN bracis-19024 294 68 chaimowiczview chaimowiczview PROPN bracis-19024 294 69 author author NOUN bracis-19024 294 70 publications publication NOUN bracis-19024 294 71 search search NOUN bracis-19024 294 72 author author NOUN bracis-19024 294 73 on on ADP bracis-19024 294 74 : : PUNCT bracis-19024 294 75 pubmed pubmed PROPN bracis-19024 294 76     SPACE bracis-19024 294 77 google google PROPN bracis-19024 294 78 scholar scholar NOUN bracis-19024 294 79 corresponding correspond VERB bracis-19024 294 80 author author NOUN bracis-19024 294 81 correspondence correspondence NOUN bracis-19024 294 82 to to ADP bracis-19024 294 83 renato renato PROPN bracis-19024 294 84 luiz luiz PROPN bracis-19024 294 85 de de PROPN bracis-19024 294 86 freitas freitas PROPN bracis-19024 294 87 cunha cunha NOUN bracis-19024 294 88 . . PUNCT bracis-19024 295 1 editor editor NOUN bracis-19024 295 2 information information NOUN bracis-19024 295 3 editors editor NOUN bracis-19024 295 4 and and CCONJ bracis-19024 295 5 affiliations affiliation NOUN bracis-19024 295 6 universidade universidade PROPN bracis-19024 295 7 federal federal PROPN bracis-19024 295 8 de de X bracis-19024 295 9 sergipe sergipe PROPN bracis-19024 295 10 , , PUNCT bracis-19024 295 11 são são NOUN bracis-19024 295 12 cristóvão cristóvão PROPN bracis-19024 295 13 , , PUNCT bracis-19024 295 14 brazil brazil PROPN bracis-19024 295 15 andré andré PROPN bracis-19024 295 16 britto britto PROPN bracis-19024 295 17 universidade universidade PROPN bracis-19024 295 18 de de PROPN bracis-19024 295 19 são são PROPN bracis-19024 295 20 paulo paulo PROPN bracis-19024 295 21 , , PUNCT bracis-19024 295 22 são são PROPN bracis-19024 295 23 paulo paulo PROPN bracis-19024 295 24 , , PUNCT bracis-19024 295 25 brazil brazil PROPN bracis-19024 295 26 karina karina PROPN bracis-19024 295 27 valdivia valdivia PROPN bracis-19024 295 28 delgado delgado VERB bracis-19024 295 29 rights right NOUN bracis-19024 295 30 and and CCONJ bracis-19024 295 31 permissions permission NOUN bracis-19024 295 32 reprints reprint NOUN bracis-19024 295 33 and and CCONJ bracis-19024 295 34 permissions permission VERB bracis-19024 295 35 copyright copyright NOUN bracis-19024 295 36 information information NOUN bracis-19024 295 37 © © PROPN bracis-19024 295 38 2021 2021 NUM bracis-19024 295 39 springer springer NOUN bracis-19024 295 40 nature nature NOUN bracis-19024 295 41 switzerland switzerland PROPN bracis-19024 295 42 ag ag PROPN bracis-19024 295 43 about about ADP bracis-19024 295 44 this this DET bracis-19024 295 45 paper paper NOUN bracis-19024 295 46 cite cite VERB bracis-19024 295 47 this this DET bracis-19024 295 48 paper paper NOUN bracis-19024 295 49 de de ADP bracis-19024 295 50 freitas freitas PROPN bracis-19024 295 51 cunha cunha NOUN bracis-19024 295 52 , , PUNCT bracis-19024 295 53 r.l r.l PROPN bracis-19024 295 54 . . PROPN bracis-19024 295 55 , , PUNCT bracis-19024 295 56 chaimowicz chaimowicz PROPN bracis-19024 295 57 , , PUNCT bracis-19024 295 58 l. l. PROPN bracis-19024 295 59 ( ( PUNCT bracis-19024 295 60 2021 2021 NUM bracis-19024 295 61 ) ) PUNCT bracis-19024 295 62 . . PUNCT bracis-19024 296 1 on on ADP bracis-19024 296 2 the the DET bracis-19024 296 3 impact impact NOUN bracis-19024 296 4 of of ADP bracis-19024 296 5 mdp mdp NOUN bracis-19024 296 6 design design NOUN bracis-19024 296 7 for for ADP bracis-19024 296 8 reinforcement reinforcement NOUN bracis-19024 296 9 learning learning NOUN bracis-19024 296 10 agents agent NOUN bracis-19024 296 11 in in ADP bracis-19024 296 12 resource resource NOUN bracis-19024 296 13 management management NOUN bracis-19024 296 14 . . PUNCT bracis-19024 297 1 in in ADP bracis-19024 297 2 : : PUNCT bracis-19024 297 3 britto britto PROPN bracis-19024 297 4 , , PUNCT bracis-19024 297 5 a. a. PROPN bracis-19024 297 6 , , PUNCT bracis-19024 297 7 valdivia valdivia PROPN bracis-19024 297 8 delgado delgado PROPN bracis-19024 297 9 , , PUNCT bracis-19024 297 10 k. k. PROPN bracis-19024 297 11 ( ( PUNCT bracis-19024 297 12 eds eds PROPN bracis-19024 297 13 ) ) PUNCT bracis-19024 297 14 intelligent intelligent ADJ bracis-19024 297 15 systems system NOUN bracis-19024 297 16 . . PUNCT bracis-19024 298 1 bracis bracis PROPN bracis-19024 298 2 2021 2021 NUM bracis-19024 298 3 . . PUNCT bracis-19024 299 1 lecture lecture NOUN bracis-19024 299 2 notes note NOUN bracis-19024 299 3 in in ADP bracis-19024 299 4 computer computer NOUN bracis-19024 299 5 science science NOUN bracis-19024 299 6 ( ( PUNCT bracis-19024 299 7 ) ) PUNCT bracis-19024 299 8 , , PUNCT bracis-19024 299 9 vol vol NOUN bracis-19024 299 10 13073 13073 NUM bracis-19024 299 11 . . PUNCT bracis-19024 300 1 springer springer NOUN bracis-19024 300 2 , , PUNCT bracis-19024 300 3 cham cham PROPN bracis-19024 300 4 . . PUNCT bracis-19024 301 1 https://doi.org/10.1007/978-3-030-91702-9_6 https://doi.org/10.1007/978-3-030-91702-9_6 PROPN bracis-19024 301 2 download download PROPN bracis-19024 301 3 citation citation NOUN bracis-19024 301 4 .ris .ris PUNCT bracis-19024 301 5 .enw .enw PROPN bracis-19024 301 6 .bib .bib PUNCT bracis-19024 302 1 doi doi PROPN bracis-19024 302 2 : : PUNCT bracis-19024 302 3 https://doi.org/10.1007/978-3-030-91702-9_6 https://doi.org/10.1007/978-3-030-91702-9_6 NUM bracis-19024 302 4 published publish VERB bracis-19024 302 5 : : PUNCT bracis-19024 302 6 28 28 NUM bracis-19024 302 7 november november PROPN bracis-19024 302 8 2021 2021 NUM bracis-19024 302 9 publisher publisher NOUN bracis-19024 302 10 name name NOUN bracis-19024 302 11 : : PUNCT bracis-19024 302 12 springer springer NOUN bracis-19024 302 13 , , PUNCT bracis-19024 302 14 cham cham PROPN bracis-19024 302 15 print print PROPN bracis-19024 302 16 isbn isbn PROPN bracis-19024 302 17 : : PUNCT bracis-19024 302 18 978 978 NUM bracis-19024 302 19 - - SYM bracis-19024 302 20 3 3 NUM bracis-19024 302 21 - - PUNCT bracis-19024 302 22 030 030 NUM bracis-19024 302 23 - - PUNCT bracis-19024 302 24 91701 91701 NUM bracis-19024 302 25 - - SYM bracis-19024 302 26 2 2 NUM bracis-19024 302 27 online online ADJ bracis-19024 302 28 isbn isbn NOUN bracis-19024 302 29 : : PUNCT bracis-19024 302 30 978 978 NUM bracis-19024 302 31 - - SYM bracis-19024 302 32 3 3 NUM bracis-19024 302 33 - - PUNCT bracis-19024 302 34 030 030 NUM bracis-19024 302 35 - - PUNCT bracis-19024 302 36 91702 91702 NUM bracis-19024 302 37 - - SYM bracis-19024 302 38 9 9 NUM bracis-19024 302 39 ebook ebook NOUN bracis-19024 302 40 packages package NOUN bracis-19024 302 41 : : PUNCT bracis-19024 302 42 computer computer NOUN bracis-19024 302 43 sciencecomputer sciencecomputer NOUN bracis-19024 302 44 science science NOUN bracis-19024 302 45 ( ( PUNCT bracis-19024 302 46 r0 r0 NOUN bracis-19024 302 47 ) ) PUNCT bracis-19024 302 48 share share VERB bracis-19024 302 49 this this DET bracis-19024 302 50 paper paper NOUN bracis-19024 302 51 anyone anyone PRON bracis-19024 302 52 you you PRON bracis-19024 302 53 share share VERB bracis-19024 302 54 the the DET bracis-19024 302 55 following follow VERB bracis-19024 302 56 link link NOUN bracis-19024 302 57 with with ADP bracis-19024 302 58 will will AUX bracis-19024 302 59 be be AUX bracis-19024 302 60 able able ADJ bracis-19024 302 61 to to PART bracis-19024 302 62 read read VERB bracis-19024 302 63 this this DET bracis-19024 302 64 content content NOUN bracis-19024 302 65 : : PUNCT bracis-19024 302 66 get get VERB bracis-19024 302 67 shareable shareable ADJ bracis-19024 302 68 linksorry linksorry NOUN bracis-19024 302 69 , , PUNCT bracis-19024 302 70 a a DET bracis-19024 302 71 shareable shareable ADJ bracis-19024 302 72 link link NOUN bracis-19024 302 73 is be AUX bracis-19024 302 74 not not PART bracis-19024 302 75 currently currently ADV bracis-19024 302 76 available available ADJ bracis-19024 302 77 for for ADP bracis-19024 302 78 this this DET bracis-19024 302 79 article article NOUN bracis-19024 302 80 . . PUNCT bracis-19024 303 1 copy copy VERB bracis-19024 303 2 shareable shareable ADJ bracis-19024 303 3 link link NOUN bracis-19024 303 4 to to PART bracis-19024 303 5 clipboard clipboard NOUN bracis-19024 303 6 provided provide VERB bracis-19024 303 7 by by ADP bracis-19024 303 8 the the DET bracis-19024 303 9 springer springer NOUN bracis-19024 303 10 nature nature PROPN bracis-19024 303 11 sharedit sharedit PROPN bracis-19024 303 12 content content NOUN bracis-19024 303 13 - - PUNCT bracis-19024 303 14 sharing share VERB bracis-19024 303 15 initiative initiative NOUN bracis-19024 303 16 keywords keyword NOUN bracis-19024 303 17 reinforcement reinforcement NOUN bracis-19024 303 18 learning learning NOUN bracis-19024 303 19 resource resource NOUN bracis-19024 303 20 management management NOUN bracis-19024 303 21 markov markov NOUN bracis-19024 303 22 decision decision NOUN bracis-19024 303 23 processes process NOUN bracis-19024 303 24 publish publish VERB bracis-19024 303 25 with with ADP bracis-19024 303 26 us us PROPN bracis-19024 303 27 policies policy NOUN bracis-19024 303 28 and and CCONJ bracis-19024 303 29 ethics ethic NOUN bracis-19024 303 30 search search NOUN bracis-19024 303 31 search search NOUN bracis-19024 303 32 by by ADP bracis-19024 303 33 keyword keyword NOUN bracis-19024 303 34 or or CCONJ bracis-19024 303 35 author author NOUN bracis-19024 303 36 search search NOUN bracis-19024 303 37 navigation navigation NOUN bracis-19024 303 38 find find VERB bracis-19024 303 39 a a DET bracis-19024 303 40 journal journal NOUN bracis-19024 303 41 publish publish VERB bracis-19024 303 42 with with ADP bracis-19024 303 43 us we PRON bracis-19024 303 44 track track VERB bracis-19024 303 45 your your PRON bracis-19024 303 46 research research NOUN bracis-19024 303 47 discover discover VERB bracis-19024 303 48 content content NOUN bracis-19024 303 49 journals journal NOUN bracis-19024 303 50 a a DET bracis-19024 303 51 - - PUNCT bracis-19024 303 52 z z NOUN bracis-19024 303 53 books book NOUN bracis-19024 303 54 a a DET bracis-19024 303 55 - - PUNCT bracis-19024 303 56 z z NOUN bracis-19024 303 57 publish publish NOUN bracis-19024 303 58 with with ADP bracis-19024 303 59 us us PROPN bracis-19024 303 60 journal journal PROPN bracis-19024 303 61 finder finder PROPN bracis-19024 303 62 publish publish VERB bracis-19024 303 63 your your PRON bracis-19024 303 64 research research NOUN bracis-19024 303 65 language language NOUN bracis-19024 303 66 editing edit VERB bracis-19024 303 67 open open ADJ bracis-19024 303 68 access access NOUN bracis-19024 303 69 publishing publishing NOUN bracis-19024 303 70 products product NOUN bracis-19024 303 71 and and CCONJ bracis-19024 303 72 services service NOUN bracis-19024 303 73 our our PRON bracis-19024 303 74 products product NOUN bracis-19024 303 75 librarians librarian VERB bracis-19024 303 76 societies society NOUN bracis-19024 303 77 partners partner NOUN bracis-19024 303 78 and and CCONJ bracis-19024 303 79 advertisers advertiser NOUN bracis-19024 303 80 our our PRON bracis-19024 303 81 brands brand NOUN bracis-19024 303 82 springer springer NOUN bracis-19024 303 83 nature nature PROPN bracis-19024 303 84 portfolio portfolio PROPN bracis-19024 303 85 bmc bmc PROPN bracis-19024 303 86 palgrave palgrave PROPN bracis-19024 303 87 macmillan macmillan PROPN bracis-19024 303 88 apress apress PROPN bracis-19024 303 89 discover discover VERB bracis-19024 303 90 your your PRON bracis-19024 303 91 privacy privacy NOUN bracis-19024 303 92 choices choice NOUN bracis-19024 303 93 / / SYM bracis-19024 303 94 manage manage NOUN bracis-19024 303 95 cookies cookie NOUN bracis-19024 303 96 your your PRON bracis-19024 303 97 us us PROPN bracis-19024 304 1 state state NOUN bracis-19024 304 2 privacy privacy NOUN bracis-19024 304 3 rights right NOUN bracis-19024 304 4 accessibility accessibility NOUN bracis-19024 304 5 statement statement NOUN bracis-19024 304 6 terms term NOUN bracis-19024 304 7 and and CCONJ bracis-19024 304 8 conditions condition NOUN bracis-19024 304 9 privacy privacy NOUN bracis-19024 304 10 policy policy NOUN bracis-19024 304 11 help help NOUN bracis-19024 304 12 and and CCONJ bracis-19024 304 13 support support VERB bracis-19024 304 14 legal legal ADJ bracis-19024 304 15 notice notice NOUN bracis-19024 304 16 cancel cancel VERB bracis-19024 304 17 contracts contract NOUN bracis-19024 304 18 here here ADV bracis-19024 304 19 129.74.145.123 129.74.145.123 NUM bracis-19024 304 20 hesburgh hesburgh PROPN bracis-19024 304 21 library library PROPN bracis-19024 304 22 er er INTJ bracis-19024 304 23 unit unit NOUN bracis-19024 304 24 ( ( PUNCT bracis-19024 304 25 3005732405 3005732405 NUM bracis-19024 304 26 ) ) PUNCT bracis-19024 304 27 northeast northeast ADJ bracis-19024 304 28 research research NOUN bracis-19024 304 29 libraries library NOUN bracis-19024 304 30 ( ( PUNCT bracis-19024 304 31 nerl nerl PROPN bracis-19024 304 32 ) ) PUNCT bracis-19024 304 33 ( ( PUNCT bracis-19024 304 34 8200828607 8200828607 NUM bracis-19024 304 35 ) ) PUNCT bracis-19024 304 36 nerl nerl VERB bracis-19024 304 37 ta ta X bracis-19024 304 38 account account NOUN bracis-19024 304 39 ( ( PUNCT bracis-19024 304 40 3006206169 3006206169 NUM bracis-19024 304 41 ) ) PUNCT bracis-19024 304 42 university university NOUN bracis-19024 304 43 of of ADP bracis-19024 304 44 notre notre PROPN bracis-19024 304 45 dame dame PROPN bracis-19024 304 46 hesburgh hesburgh PROPN bracis-19024 304 47 library library NOUN bracis-19024 304 48 ( ( PUNCT bracis-19024 304 49 3000184373 3000184373 NUM bracis-19024 304 50 ) ) PUNCT bracis-19024 305 1 © © ADP bracis-19024 305 2 2025 2025 NUM bracis-19024 305 3 springer springer NOUN bracis-19024 305 4 nature nature NOUN