The new nFL exTra poinT ruLe FirsT impLemenTed in The 2015 season requires a kicker To aTTempT his exTra poinT wiTh The baLL snapped From The 15-yard Line. This aTTempT sTreTches an exTra poinT To The equivaLenT oF a 32-yard FieLd goaL aTTempT, 13 yards Longer Than under The previous ruLe. Though a 32-yard aTTempT is sTiLL a chip shoT To any proFessionaL kicker, many nFL anaLysTs were surprised To see The number oF exTra poinTs ThaT were missed. shouLd This reaLLy have been a surprise, Though? beginning wiTh a repLicaTion oF a sTudy by cLark eT. aL, This sTudy aims To expLore The worLd oF nFL kicking From a sTaTisTicaL perspecTive, appLying economeTric and machine Learn- ing modeLs To dispLay a deeper perspecTive on whaT exacTLy makes some FieLd goaL aT- TempTs more diFFicuLT Than oThers. uLTimaTeLy, The goaL is To go beyond The previous research on This Topic, providing an improved predicTive modeL oF FieLd goaL success and a beTTer meTric For evaLuaTing pLacekicker abiLiTy. Extra point undEr rEviEw Machine Learning and the NFL Field Goal James Ledoux data selection The data used in this experiment comes from Armchair Analysis, covering every field goal attempted from the 2000 through 2013 seasons, roughly 14,000 attempts in total. It contains the binary outcomes of field goal at- tempts, atmospheric conditions at kickoff time (tempera- ture, precipitation, humidity, wind speed), and various sit- uational variables, such as kick distance, time left in the game, the game’s score, and whether the kicker was iced. While there is no way to predict probability with perfect precision, the size of this data set allows certain models to form predictions that are reasonably close to the true value on this 0-to-1 scale of make or miss. Here is an overview of the most significant information in the data set: Distance: Distance is the most important variable in this data set. The distances in the data range from 18 to 76 yards, with the inner quartile range being from 25 to 45 yards, and the median distance being 37 yards. The distribution of dis- tances in the data set is more or less uniform from 20 to 50 yards, but sparse beyond 55. Stadium Factors: The data also includes the stadium in which the kick took place, whether it took place on turf or natural grass, and whether this happened at a home or an away game. Most of these situational factors proved insignificant, and spe- cific stadiums were not included in the final models. There was, however, significant proof to support the much-dis- cussed “Mile-High Effect,” under which kickers experi- ence significantly increased range capabilities while play- ing in the high altitude of the Denver Broncos’ Mile High Stadium. Climate Factors: Among the climate factors measured were temperature, humidity, wind speed, and precipitation. The distributions of these factors are presented below: Given these distributions, I carried out similar feature en- gineering to that done in the Clark paper, creating a binary “cold” dummy variable for temperatures under 50°F, a “windy” variable for wind speeds greater than 10 mph, and a “humid” one for humidities greater than 60%. The cold and windy dummy variables are useful because both wind speeds and temperatures vary throughout games, while the measurements in this data set are only those recorded Figure 3. HuMiDiTY DisTribuTions in THe DaTa Figure 1. TeMperaTure DisTribuTions in THe DaTa Figure 2. WinD DisTribuTions in THe DaTa 56 ElEmEnts : : FAll 2016 at kickoff time. Converting these measurements to cate- gorical variables limits the number of cases that are mis- classified due to this limitation in the data set. It should be noted that I do, however, use continuous temperature as a variable in certain later models, with temperatures typi- cally being stable within a reasonable range to make the continuous variable useful despite its time-of-measure- ment flaw. Humidity was not significant in any of the mod- els, and was dropped. Also noteworthy is that the data set contained a variety of classifications for weather conditions such as “rain”, “snow”, “light rain”, “stormy”, “foggy”, “clear”, and so on. To reduce the ambiguity, I lumped all rain and snow-con- taining conditions into a single dummy variable called “precipitation.” Of this, the set has 847 instances with pre- cipitation, and 13,144 without. There were many other variables to test in the data set, but these proved to be the most important to the end-models. The individual effects of each of these variables on the final predictive models can be seen in the appendix, exhibit B. BinoMial loGistic ReGRession Models The first model I tested was a logistic regression model, replicating that used by Clark et al., and experimenting with some slight adjustments. Logistic regression is a non-linear translation of the stan- dard MLR function, measuring probability between the only two possible field goal outcomes of a make and a miss (1 and 0). This probability is calculated by the function dis- played at the bottom of this page. The Clark paper removes all regressors not statistically sig- nificant at the 5% level (P value < .05). This leads it to only include distance and a series of environmental factors, omitting all situational ones such as point spread, the kick- er being iced, the home field, and etc. I tested three mod- els, the first being a perfect replication of the Clark model, the second including the icing of the kicker, and the third including statistically significant home-field advantages. Model 1: Clark Replication My first task was to replicate the Clark study with more recent data. Clark’s model uses data from 2000-2011, so I replicated this with the 2012 and 2013 seasons included to see if there were any significant changes. The coefficients were all close to those from the original study, and each of the variables either became more significant or remained the same. The outputs are shown in Table 1 and can be compared to the originals in exhibit C. Model 2: Icing the Kicker After replicating the study, I included the variable of icing the kicker. This is a binary variable equal to one when the opposing team had called a timeout just before the field goal was attempted in a potentially game winning or tying situation, and equal to zero in all other cases. This was ex- cluded in the Clark model because it did not meet the sig- nificance level of 5%. Although this is still the case in my model, I argue that a 9.1% significance level is sufficient Table 1. Clark repliCaTion regression ouTpuTs 57 extra point under review for a binary variable such as this one, given the importance of icing kickers to the outcomes of games. This model is discussed in greater detail below. Model 3: Home Field Advantages For model 3, I regressed all of the previously mentioned variables as well as the individual stadium dummies on field goal outcomes in order to see which home fields were the most significant. I then iteratively removed the least significant stadiums until I arrived at a list that met the 10% significance level, relaxing my significance levels slightly in order to get a better view of home field advan- tages in the data set. The stadiums that were significant at the 10% level and their coefficients are: Adelphia Coliseum (-.3567), Giants Stadium (-0.5444), Network Associates Coliseum (-0.6063), Paul Brown Stadium (-0.2335), Ralph Wilson Stadium (-0.3736), Texas Stadium (-0.3096), and Veterans Stadium (-0.1536). The reason most stadiums were insignificant appears to be the high correlation between many stadiums and their associated weather conditions. Lambeau Field, Soldier Field, and Gillette Stadium, for example, were all storied stadiums with reportedly difficult crowds to play against. These stadiums also, however, have high levels of precipi- tation and historically cold temperatures. This problem with collinearity led me to discard this model altogether, with the intuition that each stadium dummy is going to be highly correlated with its home-city’s climate conditions and the type of surface of its playing field. Of the three logistic regression models tested, I found the second model, with icing the kicker included, to be the most useful. Though it does not quite meet the same stan- dard of significance as the Clark model, the importance of icing the kicker and the explanatory power it offers in game-winning and tying scenarios–one of the most entic- ing times to use such a model–makes this a worthwhile sacrifice. Additionally, one sub-10% significance level is certainly not problematic in a model such as this one, where everything else meets a high standard of proof. With this established as my model of choice, the regres- sion output is as follows: When tested against data it had not seen before, here is the layout of the model’s predictions measured against their distances: Figure 4 shows that the presence of the various environ- mental factors and the icing of the kicker creates a consid- erable spread in field goal difficulties, even within the same distance-levels. A 50-yard field goal, for example, can “Field goal kicking, however, being the isolated component of play that it is, has the potential to be one of the first great leaps in the quantification of the ways that coaches, scouts and fans view the game.” Table 2. MoDel 2 regression ouTpuT 58 ElEmEnts : : FAll 2016 have anywhere between a 41 and 72 percent probability of success given this wide array of non-distance factors. consideRinG Field Goal diFFicUltY WHen RatinG KicKeR aBilitY The most common metric used to evaluate kicker perfor- mance is “make percentage.” While this is an acceptable surface-level measurement, it makes the false assumption that all kicks are created equal, which is simply not the case. Based on team strategy and quality, some kickers may attempt a greater portion of their field goals from lon- ger distances than others. Also, due to climate factors, a kicker in New England will typically have a more difficult job than one in Oakland, all other variables held equal. Because of factors such as these, make percentage is a lacking metric; a statistic taking field goal difficulty into account would tell a deeper story. Such a metric can be formed using the logistic regression model created above. To do this, one simply takes the dif- ference between each field goal’s outcome (0 for miss, 1 for make) and its predicted probability of success, as deter- mined by the model, and multiplies this by 3, the number of points that the field goal is worth if successful. This cre- ates a “points added” metric, showing the number of points that the kicker has generated for his team above the expected value, or, in other words, the value that the kicker has created above that which would be generated by a per- fectly average replacement. This standardized-points mea- sure punishes kickers greater amounts for missing “easy” field goals than it does difficult ones, overcoming the shortcomings of using the make percentage statistic alone. Points Added = (field goal success – probability of success) x 3. This metric, measuring field goal outcomes against their difficulty level, will provide an unbiased assessment of a kicker’s skill. Who are the best kickers of the modern era? Using this statistic, one can determine not only who have been the best kickers of the modern era, but also the most underrated, taking the difference between kickers’ rank- ings according to the points-added statistic and those de- termined by make percent. Sebastian Janikowski and Stephen Gostkowski are two of the great stories of this metric. Gostkowski is widely touted as the best kicker in the game today. Much of his praise is well earned: he is consistently among the most accurate kickers in the NFL as measured by field goal percent (in the two most recent seasons not in this data set, he scored on 94.6 and 91.7% of his attempts). However, Gostkowski also plays for the high-powered New England offense, and part of his high make percentage ap- pears to result from him being set up for large amounts of easier field goals. Because of this, while he is ranked 8th in field goal percentage, he is only 23rd in points-added, mak- ing him overrated by 15 positions on the list. Granted, giv- en his successful 2014 and 2015 seasons, it is likely that these ranks have improved since 2013. Figure 4: logisTiC regression MoDel preDiCTions (probabiliTY THaT a FielD goal is suCCessFul). For a larger vieW, see exHibiT D Table 3: besT kiCkers bY poinTs aDDeD, Fg%, anD Degree unDerraTeD 59 extra point under review Janikowski, on the other hand, plays for the woeful Oak- land Raiders, who have struggled offensively throughout the past several seasons and relied greatly on Janikowski’s powerful leg during the period tested. As a result, he was set up with a disproportionately high amount of long field goal attempts, and his 80% field goal accuracy from 2000- 2013 (ranked 44th) does not tell the full story of his impact. Due to his ability to convert from 50+ yards, his points- added score is quite high at .09/attempt, making him un- derrated by 30 positions on the list. Who had the best seasons? Finally, one can also use this statistic to measure which kickers have had the greatest impacts on individual sea- sons. Taking the sum of a kicker’s points added by year gives his total points added for each season, showing ex- actly how valuable a kicker was to his team each year. Here are the greatest single-season performances according to this measurement: All of the above kickers were crucial to their teams’ offens- es these seasons. As you can see, one reason many people appear on this list is high kick quantity, which is to be ex- pected – kickers who are relied upon more heavily have increased chances of accumulating points-added for their teams. This makes it all the more impressive that Sebas- tian Janikowski’s 2009 season makes number two on this list with only 29 attempts. Although the Raiders were a losing team this season, they would have been far worse off had it not been for Janikowski bailing out their poor offense from beyond the 50 yard line multiple times over, netting an additional 19 points that the Raiders would not have secured with a replacement kicker. Table 6: WHiCH kiCkers Have THe greaTesT iMpaCTs on inDiviDiDual seasons Table 4: WHo is MosT overraTeD? (rankeD HigHer bY Fg% THan bY poinTs aDDeD) Table 5. WHo is THe MosT unDerraTeD? (rankeD HigHer bY poinTs aDDeD THan bY Fg%) 60 ElEmEnts : : FAll 2016 Neural Networks Neural Networks were the next type of model tested. This class of model loosely imitates the structure of neurons in the brain, beginning with one node per variable in the outer layer, a number of “hidden” nodes in the inner lay- ers, and then an end node for the model’s output. The model uses logistic activation functions to determine which nodes in the network will activate, and assigns weights between nodes that determine its output. The net- work uses a backpropogation algorithm to determine the optimal weights between its nodes, which allows it to im- prove the accuracy of its outputs. These models have gained a great deal of support in machine learning for their ability to gain deep insights on the patterns within data. Although I tested deeper networks, this simple 7 x 3 x 1 network proved most effective: While this was by far the most time consuming model to train and implement, it also yielded the best results of any individual model tested. This model’s success was likely due to its allowance for flexible interactions between fea- tures. Whereas a linear model such as a logistic or linear Figure 5: ranDoM ForesT FeaTure iMporTanCes Figure 6: 7 x 3 x 1 neural neTWork arCHiTeCTure iMPRoVinG PRedictions WitH MacHine leaRninG Finally, while there is a great deal to be learned from the logistic regression model and the measurements that fol- low from its predictions, I could not resist the temptation of taking this a step further to see what additional insight machine learning models could provide in this situation. Using the same data, I tested random forest, neural net- work, multiple linear regression, and hybrid models to see how they fared against the logistic regression benchmark. Random Forests A “random forest” is an ensemble-learning algorithm that uses a resampling method called “bagging” to take a single data set and break it into several overlapping sets. With one data set now broken into many, the algorithm can train an entire “forest” of decision trees rather than just one, and uses the output of each tree as a “vote” toward what the forest’s output should be. By taking the average of the many “decision trees” outputs, the random forest mod- el achieves a lower model variance than an individual deci- sion tree while maintaining a similar level of bias, mean- ing that it is both more stable and equally accurate. In this case, however, the random forest underperformed the other models tested, and did not prove useful for predic- tion improvement. This may be due to a combination of unreliable measurements in the explanatory variables and the fact that the majority of explanatory variables were cat- egorical rather than continuous, restricting the decision breaks available to the forest. Additional methods that could be tested for improving this model are breaking con- tinuous variables into buckets rather than binary variables, and further tuning the parameters for tree depth and num- ber of observations required per leaf node. 61 extra point under review regression applies a single function to transform input fea- tures into outputs, the neural network has intermediary hidden layers that allow features to interact before being transformed into an output. The result is a model that learns its own features, as opposed to other models which are constrained to the features that a user feeds them. The slow learning of this model, however, poses a significant problem for testing and improvement, making parameter- -tuning and model evaluation methods such as cross vali- dation time consuming in the absence of improved com- puting resources. Multiple Linear Regression Finally, I tested a multiple linear regression model. Be- cause of the logistic nature of the relationship between dis- tance and field goal difficulty (the difference between the difficulty of a 55- and 65-yard field goal difficulty is much larger than that between a 25 and 35-yard attempt), simply throwing all the above variables into a standard MLR mod- el will not fit the data well. Adding the log of the distance into a model with all of the previously used variables, how- ever, yielded quality results, with an MSE measurement beating that of the random forest model and the original Clark logistic model. This MLR model allows for the non- distance variables to be linear intercept variables, and also makes for the highest ease of interpretation of all the mod- els used, using the standard MLR model at the bottom of the page. Table 7 shows the regression output. Different Models, Different Predictions It should not come as a surprise that different models yield slightly different predictions. The regression outputs and error measurements mentioned up until this point in the paper, however, do not display model patterns well. Be- cause distance is such a strong predictive factor in this par- ticular set of models, simply showing probabilities graphed against distance seems to do the trick for illustrating prob- ability patterns. The final models are displayed in Figures 7, 8, 9, and 10. And the winner is... While logistic regression is the standard for probabilistic measurements such as this, it was not the most precise of those tested. The best performing model in this situation is actually one that has not been mentioned yet. While the neural network performed best among all the models al- Table 7: Mlr MoDel ouTpuT Figure 7 (Top): ranDoM ForesT Figure 8 (boTToM): neural neTWork 62 ElEmEnts : : FAll 2016 Figure 9: MulTiple linear regression Figure 10: logisTiC regression ready discussed, the best performer was one that com- bined the results of this and the second logistic regression model, taking the average of the predictions of the two. Table 8 shows how all the models compared in terms of MSEs (mean squared errors). MSE is an error measurement taking the average squared distance between the predicted values (from the model) and their true values (1 if made, 0 if missed) across all pre- dictions made. The smaller the MSE score, the more ac- curate the model. The error measurements in this case offer little descriptive significance as there is no true meth- od for evaluating the accuracy of an observation’s probabil- ity estimate. The scores serve merely to rank the models against one another. Table 8: Mse oF DiFFernT MoDels While the neural network, MLR model, and logistic regres- sion model with icing included were able to beat the logis- tic regression benchmark set by Clark, the best model of all was that which combined the predictions of the top two performers. WHY does tHis MatteR? I present two scenarios to illustrate how the analysis pre- sented may be relevant to decision-making in NFL games. The first took place on January 19, 2002, when the New England Patriots were playing the Oakland Raiders in the AFC divisional playoff game. The Patriots were down 13-10 when Adam Vinatieri lined up for a 45-yard attempt with time running low. In 19-degree weather and heavy snow, Vinatieri’s kick went through the uprights with only 27 seconds left, and the Patriots went on to win the game in overtime on another Vinatieri field goal. This snowy field goal has gone down in NFL history as one of the greatest clutch kicks of all time. It also happens to be an excellent example of one of the worst possible condi- tions to kick a field goal in, showing the need for models such as those discussed in this paper. The probabilities of Vinatieri’s kick going in had the game been in Oakland vs. in New England are illustrated in Figure 11. Had the game taken place in Oakland with its temperate climate, Vinatieri’s 45-yard attempt would have had a very reasonable 71% likelihood of succeeding. In the New Eng- land blizzard, however, it had much riskier odds, at 54%. Though the kick was ultimately good, this model goes to show just how high the stakes were that Sunday afternoon, and the amount of pressure that Vinatieri, the coaching 63 extra point under review staff, and the New England fan base must have felt as they waited for the result. The second situation is one that has not happened yet, but one that every NFL coach should be concerned about: a kicker’s missed extra point costing his team the game. Moving the extra point back 13 yards may still seem like an easy conversion, being a 32-yard attempt, but as these models show, there is a noticeable decline in the probabil- ity of success in going from a 19 to a 32-yard kick. The re- sult of this decreased likelihood was seen across the league this season, with extra point conversion percentages de- creasing from 99.6 to roughly 95 percent with the imple- mentation of the new rule. While nobody has lost a game due to this rule change just yet, this new area of risk is a perfect example of the importance of understanding field goal likelihoods for making effective coaching decisions. This increased risk will make two-point conversion at- tempts increasingly appropriate in the NFL. As is seen in these two scenarios, understanding field goal conversion likelihoods is crucial to a coach’s decision-mak- ing process. Understanding not only that distance, climate factors, and situational pressure have tangible impacts on field goal success likelihoods, but also to what extent they do, could make all the difference in weighing the risk and potential reward of a crucial coaching decision. Figure 11: iMpaCT oF WeaTHer ConDiTions on THe snoW boWl gaMe. To conclude, this study has two main findings: First, that field goals follow predictable patterns, allowing their probabilities of success to be modeled to a useful de- gree. This can and should be taken into consideration by coaching staffs during their evaluations of game time deci- sions. Second, that good kickers really do matter. Finding a great one like Rob Bironas or Sebastian Janikowski can make all the difference in a 16-game season where playoff outcomes are not determined until the very last week, or in a playoff game where a single point-added above average replace- ment is all that stands between being a runner-up and a Super Bowl champion. While baseball has been thoroughly transformed by saber- metrics and the Moneyball revolution, football still has a way to go in its acceptance of statistics as the governing dynamics of certain aspects of play, given its smaller sam- ple sizes because of its 16-game seasons. Field goal kick- ing, however, being the isolated component of play that it is, has the potential to be one of the first great leaps in the quantification of the ways that coaches, scouts and fans view the game. Through the acceptance of methods and models such as those discussed in this paper, NFL organi- zations can make great strides in improving their abilities to make data-informed decisions and maximize the poten- tial values of their rosters. gaMe aT oak gaMe aT ne64 ElEmEnts : : FAll 2016 exHibiT a: league-WiDe Fg% bY DisTanCe, 2000-2013 exHibiT b1: eFFeCTs oF paraMeTer CHanges — WinD speeD eFFeCT exHibiT b2: eFFeCTs oF paraMeTer CHanges — TurF eFFeCT exHibiT b3: eFFeCTs oF paraMeTer CHanges — preCipiTaTion eFFeCT aPPendiX TurF noT TurF noT WinDY WinDY no preCipiTaTion preCipiTaTion 65 extra point under review exHibiT b4: eFFeCTs oF paraMeTer CHanges — iCing THe kiCker exHibiT b5: eFFeCTs oF paraMeTer CHanges — Mile- HigH eFFeCT exHibiT b6: eFFeCTs oF paraMeTer CHanges — ColD WeaTHer eFFeCT exHibiT C: original sTuDY CoeFFiCienTs. iCeD noT iCeD ColD noT ColD plaYeD elseWHere gaMe plaYeD in Denver 66 ElEmEnts : : FAll 2016 exHibiT D: probabiliTY preDiCTions FroM logisTiC MoDel 2 endnotes 1. Torin, Johnson, and Stimpson, “Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression.” 2. Affordable NFL Data Plans, Armchair Analysis. 3. Brian, “Altitude and Field Goals.” 4. Torin, Johnson, and Stimpson, “Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression.” NFL.com.,”Stephen Gostkowski.” 5. Matt, “A Step by Step Backpropagation Example,” March 17, 2015. ReFeRences Clark, Torin, Aaron Johnson, and Alexander Stimpson. “Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression.” MIT Sloan Sports Ana- lytics Conference (2013). Accessed January 11, 2016. Affordable NFL Data Plans. Accessed 11 January 2016. Armchair Analysis. Burke, Brian. “Altitude and Field Goals.” Advanced Foot- ball Analytics. Accessed January 11, 2016. “Stephen Gostkowski.” NFL.com. Accessed January 11, 2016. Mazur, Matt. “A Step by Step Backpropagation Example.” March 17, 2015. Accessed January 11, 2016. 67 extra point under review