ISSN 2744-1741 Defense and Security Studies Original Research Vol. 3, January 2022, pp.83-100 https://doi.org/10.37868/dss.v3.id199 This work is licensed under a Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) that allows others to share and adapt the material for any purpose (even commercially), in any medium with an acknowledgement of the work's authorship and initial publication in this journal. 83 Using Data Envelopment Analysis (DEA) for measuring efficiency in the Defense Sector Ivan Okromtchedlishvili1* 1 PhD student, Business Administration, Faculty of Business and Technology, International Black Sea University, Tbilisi, Georgia *Corresponding author E-mail: iokro@yahoo.com Received Oct. 3, 2022 Revised Nov. 15, 2022 Accepted Nov. 17, 2022 Abstract The way to improve the efficiency and effectiveness of public spending, which is a top priority for any government in any country, implies the introduction of Performance-Based Budgeting (PBB). One of the more advanced government- wide performance budgeting systems that uses performance information systematically in the preparation of the government budget is program budgeting. It is important to keep in mind that without systematic development and use of program performance information and adequate and effective performance indicators, program budgeting in the defense sector does not make sense as a tool to improve the efficiency and effectiveness of the defense resource management process. Only by defining and tracking success can it be known if the defense organizations and units perform efficiently and effectively. In this article, Data Envelopment Analysis (DEA) was considered an instrument that can be used to measure, evaluate, and analyze the efficiency of the state and government as a whole, as well as commercial and non-profit organizations, including military units. It can be used as an instrument to hold managers accountable for their performance, which is critical to effective PBB. In this article, DEA has been applied to NATO members and some Eastern Europe post-Soviet aspirant and partner countries (Ukraine, Georgia and Moldova) to understand how efficient each country is at achieving its military power. In order to demonstrate the feasibility of using DEA to examine the efficiency of the infantry battalions of the infantry brigades under the Eastern and Western Commands of the Georgian Defense forces (GDF), an illustrative analysis of the efficiency of the aforementioned units was carried out using fictitious data. © The Author 2022. Published by ARDA. Keywords: Program budgeting; Performance; Efficiency; Effectiveness; Data Envelopment Analysis. 1. Introduction The introduction of Performance-Based Budgeting (PBB) is a way to improve the efficiency and effectiveness of public spending, which is a top priority for any government in any country. This approach implies focusing rather on the results of spending and the achievement of policy objectives than on the management of inputs and provides budget decision-makers with greater discretion in the use of resources and in deciding the input DSS Vol. 3, January 2022, pp.83-100 84 mix. However, simultaneously, increased flexibility, and weakened central controls are counterbalanced by stronger internal controls and oversight and accountability mechanisms to hold managers accountable for the results of their performance. PBB implies the systematic use of performance information in the budget process to make results a central determinant of budget funding decisions, and thereby make budgeting a powerful instrument for maximizing government’s efficiency and effectiveness. Program budgeting is one of the most advanced nationwide performance budgeting systems that systematically uses performance information in the preparation of the state budget where expenditures are classified into groups of similar activities or projects (i.e. programs) with common outputs and outcomes. The main differences between the traditional line-item budget method and the program budget method are presented in the Table 1. Table 1. The main differences between the traditional line-item budget method and the program budget method [26] Budget Method Characteristics Primary Organization Feature Budget Focus Line-Item Expenditure Budget Expenditure by commodity or resource purchased Resources Purchased Control of Resources Program Budget Expenditure Related to Public Goals Cost data across organizational lines Achievements (products or outputs) Planning Defense is an important part of the public sector, and its organizations consume large amounts of public resources. Improving the efficiency and effectiveness of managing defense funds and ensuring a successful defense budgeting process implies introducing the performance-based (program) budgeting approach in the defense sector as well. The systematic development and use of program performance information is critical to achieving a good defense program budget. Without adequate and effective performance indicators and their application to assess the performance of program managers, program budgeting in the defense sector does not make sense as a tool to improve the efficiency and effectiveness of the defense resource management process; this will only make it easier and simpler for the ministry to allocate and use budgetary resources by loosening line item controls, without obtaining the core benefits of program budgeting. Van Dooren et al. [1, p. 20] argue that “performance can be defined as outputs and outcomes.” The principal tool of program budgeting identified by Robinson [2, p. 2], along with “budgetary expenditure classification in terms of outcome/output groups (“programs”), is “the systematic gathering of performance information (through indicators, evaluation, etc.) to inform decisions about budgetary priorities between competing programs.” Performance-based budgets require information on inputs (measured in monetary terms), outputs (units of output), efficiency and productivity data (cost per activity), and effectiveness information (level of goal achievement) [3]. In the case of the defense ministry, performance information (outputs, outcomes, and indicators) and its systematic development and use are critical to achieving a good defense program budget and an effective resource allocation. The existence of a clear linkage between resource allocation and desired/produced outputs and outcomes is crucial for defense decision-makers to provide them with the ability to compare the costs and benefits of alternative spending options and choose the most effective ones, as well as monitor and control performance. The old management adage “you can’t manage what you can’t measure” applies to the defense sector as well. Without defining and tracking success, it is impossible to know if the defense organizations and units are successful. When considering the defense sector's activities, two aspects can be distinguished. The first concerns the products/services (outputs) produced by defense entities through the use of resources and is related to DSS Vol. 3, January 2022, pp.83-100 85 efficiency ("doing things right"). The second aspect, which concerns the impact of the produced products or services (outputs) on the objectives set for defense, is related to effectiveness (“doing the right things”). As Webb & Angelis [24, p.21] noted, “to measure efficiency, we must understand the relationship between the cost of inputs and the amount of outputs […] to measure effectiveness, we must understand the relationship between the organization’s goals and objectives [or outcomes] and its outputs […].” In order to do the right things (or to achieve effectiveness), defense policy-makers and decision-makers have to choose and develop the right mix of subprogram (intermediate) outputs to produce the final output (military capability) of the defense program, maximizing their preference value for outcomes, while subprogram managers have to do things right when responsible for producing outputs efficiently [13]. Data Envelopment Analysis (DEA) can be seen as a tool to measure and evaluate the effectiveness and efficiency of the state and government as a whole, as well as non-profit and commercial organizations, units and subunits (including defense organizations and units). It can be used as an instrument to hold managers accountable for their performance which is critical to effective PBB. The results of the DEA analysis, in the form of potential improvements, offer management (commanders) opportunities to explore in search of higher performance. The process includes identifying the main sources of inefficiency, as well as those units that can become a benchmark for others. In this study, DEA has been applied to NATO members and some Eastern Europe post-Soviet aspirant and partner countries (Ukraine, Georgia and Moldova) to understand how efficient each country is at achieving its military power. The efficiency of the decision-making units was measured with a CCR model. As for the Georgian Defense Forces (GDF) units, in particular the infantry battalions of the infantry brigades under the Eastern and Western Commands, there are significant limitations in conducting such research due to the secrecy of detailed information, especially regarding the output of the defense program, namely, the military capability and its indicators – readiness levels of units. In addition, due to the peculiarities of the current defense program structure, obtaining accurate cost information, such as detailed information on the cost per battalion for any given period, should be very problematic. Therefore, in order to demonstrate the feasibility of using DEA to examine the efficiency of operational units, an illustrative analysis of the efficiency of the aforementioned infantry battalions was carried out using fictitious data. 2. Data Envelopment Analysis (DEA) 2.1 The use of the DEA in measuring performance An excellent mathematical programming tool that can be used to measure, evaluate, and analyze performance is data envelopment analysis (DEA), which has been used to evaluate the performance of many different types of organizations, including government, not-for-profit and commercial units and subunits since it was first introduced in the late 1970s. As a comparative performance measurement tool, DEA is aimed at facilitating “a program to improve performance, not to provide a simple grading of service providers” [4]. According to Avkiran, DEA is a non- parametric method that provides a comparative ratio of weighted outputs to inputs for each decision-making unit (DMU), i.e., a relative efficiency score, which is usually reported as a number from 0 to 100% or 0 to 1. A unit scoring less than 100% is considered inefficient compared to other units in the sample [5]. Efficiency can be defined as a "degree to which the observed use of resources to produce outputs of a given quality matches the optimal use of resources to produce outputs of a given quality" [4, p. 14]. Sherman defines efficiency as "the ability to produce products or services with the minimum level of resources required" [6, p. 3]. Farrell recognized the importance of measuring the extent to which outputs could be increased through higher efficiency without the use of additional inputs [7]. The Pareto optimality condition for efficient production states that a DMU is inefficient if the output can be increased without increasing any input and without decreasing any other output; likewise, a DMU is inefficient if the input can be decreased without decreasing any output and without increasing any other input [8]. DSS Vol. 3, January 2022, pp.83-100 86 DEA measures the efficiency of decision-making units (DMUs) using linear programming techniques, envelops observed input-output vectors as tightly as possible and allows to consider multiple input-output vectors at the same time without any assumptions on data distribution. In each case, efficiency is measured by proportional changes in inputs or outputs. According to Ji and Lee, “a DEA model can be subdivided into an input-oriented model, which minimizes inputs while satisfying at least the given output levels, and an output- oriented model, which maximizes outputs without requiring more of any observed input values” [9, p. 268]. The original Charnes, Cooper, and Rhodes (CCR) DEA model utilizes linear programming to produce an efficiency measure for a DMU, requiring only that the DMUs convert similar inputs to similar outputs and that these can be quantified. The logic of the model implies, first, defining the underlying premise that efficiency is the sum of weighted outputs over the sum of weighted inputs [10]. The DEA model for the kth DMU can be formulated as follows: t ∑ UrYrk Max Ek = r = 1 m ∑ ViXik i = 1 t ∑ UrYrj s.t. r = 1 ≤ 1 j = 1,……n m ∑ ViXij i = 1 Ur ≥ 0 r = 1,…….t Vi ≥ 0 i = 1,…….m, Where: Objective function Ek = the efficiency index of the k th DMU; Parameters yrj = the amount of the r th output for the jth DMU; xij = the amount of the i th input for the jth DMU; t = the number of outputs; m = the number of inputs; and n = the number of DMUs Decision variables ur = the weight assigned to the r th output; and vi = the weight assigned to the i th input [11]. Generally, in the defense sector area, DEA efficiency and productivity studies have focused on various support functions such as maintenance and recruitment, as well as operational units, the core area of defense (see Table 2). DSS Vol. 3, January 2022, pp.83-100 87 Table 2. Bibliography of DEA in the military [12] Paper Field Inputs Outputs Observa tions Lewin and Morey (1981) Recruitment 10 2 43 Charnes et al. (1985) Maintenance 8 4 42 Bowlin (1987) Maintenance 3 4 21 Bowlin (1989) Accounting and finance 1 5 18 Ali et al. (1989)* Recruitment n/a n/a n/a Roll et al. (1989) Maintenance 3 2 10-35 Clarke (1992) Maintenance 4 2 17 Ozcan and Bannick (1994) Hospitals 6 2 23 Bowlin (2004) Civil reserve air fleet 4 7 37-111 Brockett et al. (2004) Recruitment 1 10 n/a Sun (2004) Maintenance 6 5 30 Farris et al. (2006) Engineering design projects 4 1 15 Lu (2011) Kalin (2021) Hanson (2012) Hanson (2019) Outlets Logistics Operational units Combat units 4 4 3 3 (1) 2 1 1 1 (1) 31 34 11 12 *Paper not available. Hanson conducted interesting research that examined the productivity and efficiency of the core area of the Norwegian armed forces, operational units, using Data Envelopment Analysis (DEA). A model has been developed to analyze the productivity and efficiency by DEA for the operational units of the armed forces. As Hanson noted, “By aggregating activity standards and quality measures the model enables a meaningful and measurable expression for the output of an operational unit” [12, p. 25]. In another study, Hanson used a scenario-based planning approach to develop an effectiveness measurement model for situations where traditional methods such as two-stage regression fail due to long time lags and lack of variation in the variables. According to Hanson, “from a sample of 12 combat units in the Norwegian Armed Forces, producing different outputs, [it was found] that inefficiencies in output mix can explain most of the changes in overall effectiveness over a four-year period of time” [13, p. 1]. 2.2 Examining the efficiency of NATO members and some partner countries in achieving military power by using DEA DEA has been applied to NATO members and some Eastern Europe post-Soviet aspirant and partner countries (Ukraine, Georgia and Moldova) to understand how efficient each country is at achieving its military power. The efficiency of the decision-making units was measured with a CCR model. 2.2.1 Specification of Data and Variables A total of 31 DMUs were selected for the study. These DMUs focused on NATO members and some Eastern Europe post-Soviet aspirant and partner countries (Ukraine, Georgia and Moldova). The study was aimed at measuring how efficiently each country achieved its military power. Four variables were retrieved from the open sources [28], [29], [30], [31]: two input variables - Defense Expenditure, Current prices and exchange rates US dollars for 2020; Defense Expenditure as a share of real GDP (%) for 2020; and two output variables - Military Personnel for 2020; and Military Strength Ranking for 2022 (reversed). DSS Vol. 3, January 2022, pp.83-100 88 Defense expenditure refers to all current and capital spending on the armed forces of a state and in theory, the higher the level of defense expenditure, the better the military power of the state. The commitment made in 2014 by members of the North Atlantic Treaty Organization (NATO) to increase their Defense expenditure as a share of real GDP to 2 % by 2024 is still the subject of debate about military spending in NATO [25]. In my recent article, I proposed to define the main output of a defense program as “Military Capability as a comprehensive force structure consisting of its constituent force elements/capabilities […] with an integrated set of aspects categorized as doctrine, organization, training, materiel, leadership development, personnel, and facilities, and with an appropriate readiness level assessed at a concrete time” [23, p. 94]. Therefore, in this case, it is crucial that Military Personnel be considered as an integral part of military capability (combat-ready forces), i.e., the “production” of military personnel implies the simultaneous development of the military capability as a whole (across the entire DOTMLPF spectrum). In principle, the GFP rating (Military Strength Ranking) is a kind of indicator of the performance of the defense organization of a state, and an improved position in the ranking is evidence of increased efficiency and effectiveness of the defense programs. For this study, the reverse military power index (Global Firepower, 2022) was used, so it was assumed that the higher the military power index, the better the country. At the initial measure, the lower the military index, the better. 2.2.2 DEA Model and Results The DEA analysis was carried out using DEA-Solver-PRO 5.0 software [22] developed by W.W. Cooper, L.M. Seiford and K. Tone. This study performed a CCR input-oriented DEA model, and the main focus was to see how efficient each country was at producing military capability and achieving its military power index, given its resources or inputs. Table 3 and Table 4 below show the results of running the model. Table 3. DEA Test Results Rank DMU Score Rank DMU Score 1 Moldova 1 17 Netherlands 0,57707535 1 United States 1 18 Bulgaria 0,570603056 1 Spain 1 19 Poland 0,569492872 1 Türkiye 1 20 Hungary 0,55453459 5 Czech Republic 0,971773347 21 United Kingdom 0,548049942 6 Ukraine 0,932569009 22 Belgium 0,533707754 7 Italy 0,909221694 23 Croatia 0,528911822 8 Portugal 0,8447062 24 Slovenia 0,5158751 9 Georgia 0,792712503 25 Slovak Republic 0,476957309 10 France 0,697923222 26 Albania 0,376089889 11 Romania 0,665795625 27 North Macedonia 0,368317769 12 Germany 0,662166375 28 Lithuania 0,28768714 13 Greece 0,634367881 29 Latvia 0,20825904 14 Norway 0,631334284 30 Montenegro 0,197800912 15 Canada 0,610115983 31 Estonia 0,157514612 16 Denmark 0,578313316 DSS Vol. 3, January 2022, pp.83-100 89 Table 4. Projections by the CCR Model No DMU Score I/O Data Projection Difference % 1 United States 1 Defense Expenditure 7,84952E+11 7,84952E+11 0 0,00% Defense Expenditure as a share of real GDP (%) 3,72 3,72 0 0,00% Military Personnel 1346000 1346000 0 0,00% Military Strength Ranking (reversed) 35,8958 35,8958 0 0,00% 2 France 0,697923222 Defense Expenditure 52727000000 36799397750 -15927602250 -30,21% Defense Expenditure as a share of real GDP (%) 2,03 1,416784142 -0,613215858 -30,21% Military Personnel 208000 208000 0 0,00% Military Strength Ranking (reversed) 5,7275 5,7275 0 0,00% 3 United Kingdom 0,548049942 Defense Expenditure 61925000000 33937992687 -27987007313 -45,20% Defense Expenditure as a share of real GDP (%) 2,29 1,255034368 -1,034965632 -45,20% Military Personnel 156200 175283,8318 19083,83183 12,22% Military Strength Ranking (reversed) 5,2184 5,2184 0 0,00% 4 Italy 0,909221694 Defense Expenditure 26071000000 23704318784 -2366681216 -9,08% Defense Expenditure as a share of real GDP (%) 1,38 1,254725938 -0,125274062 -9,08% Military Personnel 175500 175500 0 0,00% Military Strength Ranking (reversed) 4,7871 4,7871 0 0,00% 5 Türkiye 1 Defense Expenditure 13396000000 13396000000 0 0,00% Defense Expenditure as a share of real GDP (%) 1,86 1,86 0 0,00% Military Personnel 437200 437200 0 0,00% Military Strength Ranking (reversed) 4,4351 4,4351 0 0,00% 6 Germany 0,662166375 Defense Expenditure 58902000000 39002923817 -19899076183 -33,78% Defense Expenditure as a share of real GDP (%) 1,55 1,026357881 -0,523642119 -33,78% Military Personnel 186900 186900 0 0,00% Military Strength Ranking (reversed) 4,3067 4,3067 0 0,00% 7 Spain 1 Defense Expenditure 12828000000 12828000000 0 0,00% Defense Expenditure as a share of real GDP (%) 1 1 0 0,00% Military Personnel 122500 122500 0 0,00% Military Strength Ranking (reversed) 3,7337 3,7337 0 0,00% 8 Ukraine 0,932569009 Defense Expenditure 5924000000 5524538808 -399461191,8 -6,74% DSS Vol. 3, January 2022, pp.83-100 90 Defense Expenditure as a share of real GDP (%) 4,1 3,823532936 -0,276467064 -6,74% Military Personnel 209000 209000 0 0,00% Military Strength Ranking (reversed) 3,4016 6,261898544 2,860298544 84,09% 9 Canada 0,610115983 Defense Expenditure 23595000000 14395686628 -9199313372 -38,99% Defense Expenditure as a share of real GDP (%) 1,44 0,878567016 -0,561432984 -38,99% Military Personnel 71000 111398,8073 40398,80729 56,90% Military Strength Ranking (reversed) 3,3736 3,3736 0 0,00% 10 Poland 0,569492872 Defense Expenditure 13590000000 7739408124 -5850591876 -43,05% Defense Expenditure as a share of real GDP (%) 2,28 1,298443747 -0,981556253 -43,05% Military Personnel 120000 120000 0 0,00% Military Strength Ranking (reversed) 3,1759 3,1759 0 0,00% 11 Greece 0,634367881 Defense Expenditure 5019000000 3183892393 -1835107607 -36,56% Defense Expenditure as a share of real GDP (%) 2,65 1,681074884 -0,968925116 -36,56% Military Personnel 107600 107600 0 0,00% Military Strength Ranking (reversed) 2,8681 2,8681 0 0,00% 12 Norway 0,631334284 Defense Expenditure 7272000000 4591062910 -2680937090 -36,87% Defense Expenditure as a share of real GDP (%) 2,01 1,26898191 -0,74101809 -36,87% Military Personnel 20800 54698,49719 33898,49719 162,97 % Military Strength Ranking (reversed) 2,6527 2,6527 0 0,00% 13 Netherlands 0,57707535 Defense Expenditure 13125000000 7574113967 -5550886033 -42,29% Defense Expenditure as a share of real GDP (%) 1,47 0,848300764 -0,621699236 -42,29% Military Personnel 40000 75401,12632 35401,12632 88,50% Military Strength Ranking (reversed) 2,5771 2,5771 0 0,00% 14 Romania 0,665795625 Defense Expenditure 5051000000 3362933704 -1688066296 -33,42% Defense Expenditure as a share of real GDP (%) 2,03 1,351565119 -0,678434881 -33,42% Military Personnel 66400 66400 0 0,00% Military Strength Ranking (reversed) 2,5072 2,5072 0 0,00% 15 Czech Republic 0,971773347 Defense Expenditure 3201000000 3110646484 -90353515,56 -2,82% Defense Expenditure as a share of real GDP (%) 1,31 1,273023085 -3,70E-02 -2,82% Military Personnel 26800 41984,69203 15184,69203 56,66% Military Strength Ranking (reversed) 2,3944 2,3944 0 0,00% 16 Portugal 0,8447062 DSS Vol. 3, January 2022, pp.83-100 91 Defense Expenditure 3306000000 2792598697 -513401303 -15,53% Defense Expenditure as a share of real GDP (%) 1,43 1,207929866 -0,222070134 -15,53% Military Personnel 28700 38467,306 9767,306 34,03% Military Strength Ranking (reversed) 2,2436 2,2436 0 0,00% 17 Hungary 0,55453459 Defense Expenditure 2770000000 1536060814 -1233939186 -44,55% Defense Expenditure as a share of real GDP (%) 1,79 0,992616916 -0,797383084 -44,55% Military Personnel 22700 25069,63378 2369,63378 10,44% Military Strength Ranking (reversed) 1,7083 1,7083 0 0,00% 18 Denmark 0,578313316 Defense Expenditure 4979000000 2879422001 -2099577999 -42,17% Defense Expenditure as a share of real GDP (%) 1,4 0,809638643 -0,590361357 -42,17% Military Personnel 18100 34469,74292 16369,74292 90,44% Military Strength Ranking (reversed) 1,6836 1,6836 0 0,00% 19 Slovak Republic 0,476957309 Defense Expenditure 2050000000 977762482,9 -1072237517 -52,30% Defense Expenditure as a share of real GDP (%) 1,96 0,934836325 -1,025163675 -52,30% Military Personnel 12900 19568,30014 6668,300138 51,69% Military Strength Ranking (reversed) 1,5252 1,5252 0 0,00% 20 Croatia 0,528911822 Defense Expenditure 1031000000 545308088,2 -485691911,8 -47,11% Defense Expenditure as a share of real GDP (%) 1,8 0,952041279 -0,847958721 -47,11% Military Personnel 15200 16045,33197 845,3319659 5,56% Military Strength Ranking (reversed) 1,4729 1,4729 0 0,00% 21 Bulgaria 0,570603056 Defense Expenditure 1075000000 613398285,7 -461601714,3 -42,94% Defense Expenditure as a share of real GDP (%) 1,55 0,884434738 -0,665565262 -42,94% Military Personnel 25600 25600 0 0,00% Military Strength Ranking (reversed) 1,3664 1,3664 0 0,00% 22 Belgium 0,533707754 Defense Expenditure 5427000000 2896431982 -2530568018 -46,63% Defense Expenditure as a share of real GDP (%) 1,05 0,560393142 -0,489606858 -46,63% Military Personnel 25200 31646,37661 6446,376605 25,58% Military Strength Ranking (reversed) 1,3265 1,3265 0 0,00% 23 Lithuania 0,28768714 Defense Expenditure 1176000000 338320076,6 -837679923,4 -71,23% Defense Expenditure as a share of real GDP (%) 2,11 0,607019865 -1,502980135 -71,23% Military Personnel 16300 16300 0 0,00% Military Strength Ranking (reversed) 0,8633 0,924254558 6,10E-02 7,06% 24 Slovenia 0,5158751 DSS Vol. 3, January 2022, pp.83-100 92 Defense Expenditure 568000000 293017056,6 -274982943,4 -48,41% Defense Expenditure as a share of real GDP (%) 1,08 0,557145108 -0,522854892 -48,41% Military Personnel 7000 9164,882239 2164,882239 30,93% Military Strength Ranking (reversed) 0,8573 0,8573 0 0,00% 25 Georgia 0,792712503 Defense Expenditure 292000000 231472050,9 -60527949,13 -20,73% Defense Expenditure as a share of real GDP (%) 1,8 1,426882505 -0,373117495 -20,73% Military Personnel 20650 20650 0 0,00% Military Strength Ranking (reversed) 0,8169 2,09945964 1,28255964 157,00 % 26 Latvia 0,20825904 Defense Expenditure 743000000 154736466,7 -588263533,3 -79,17% Defense Expenditure as a share of real GDP (%) 2,22 0,462335069 -1,757664931 -79,17% Military Personnel 7000 7000 0 0,00% Military Strength Ranking (reversed) 0,6953 0,6953 0 0,00% 27 Moldova 1 Defense Expenditure 44500000 44500000 0 0,00% Defense Expenditure as a share of real GDP (%) 0,4 0,4 0 0,00% Military Personnel 5150 5150 0 0,00% Military Strength Ranking (reversed) 0,5859 0,5859 0 0,00% 28 Estonia 0,157514612 Defense Expenditure 719000000 113253006,3 -605746993,7 -84,25% Defense Expenditure as a share of real GDP (%) 2,32 0,365433901 -1,954566099 -84,25% Military Personnel 6600 6600 0 0,00% Military Strength Ranking (reversed) 0,5455 0,5455 0 0,00% 29 Albania 0,376089889 Defense Expenditure 188000000 70704899,2 -117295100,8 -62,39% Defense Expenditure as a share of real GDP (%) 1,27 0,47763416 -0,79236584 -62,39% Military Personnel 6700 6700 0 0,00% Military Strength Ranking (reversed) 0,4276 0,701893273 0,274293273 64,15% 30 Montenegro 0,197800912 Defense Expenditure 83000000 16417475,73 -66582524,27 -80,22% Defense Expenditure as a share of real GDP (%) 1,73 0,147572816 -1,582427184 -91,47% Military Personnel 1900 1900 0 0,00% Military Strength Ranking (reversed) 0,1695 0,216157282 0,046657282 27,53% 31 North Macedonia 0,368317769 Defense Expenditure 154000000 56720936,47 -97279063,53 -63,17% Defense Expenditure as a share of real GDP (%) 1,25 0,460397212 -0,789602788 -63,17% Military Personnel 6100 6100 0 0,00% Military Strength Ranking (reversed) 0,1283 0,67508041 0,54678041 426,1% DSS Vol. 3, January 2022, pp.83-100 93 2.2.3 Summary and Recommendations The study showed that some states were efficient at producing their military capability and achieving the military power index utilizing the inputs provided, while others needed to improve. According to the test results, 4 out of the 31 countries (United States, Spain, Türkiye and Moldova) outputted 1.00 or 100 percent efficiency across the DEA model and can be used as benchmarks. The CCR input-oriented model we applied measures technical efficiency, or how resources are used during the production/delivery of an output (doing the things right). Since the scores of other states were below 1.00 or 100 percent, this means that the ministries of defense should look for ways to utilize the allocated budgetary resources more efficiently to achieve higher results and improve positions in the Military Strength Ranking. 2.3 A sample DEA model for measuring the efficiency of the GDF infantry battalions In the case of the Georgian Defense Forces (GDF), the DEA can be used, for example, to examine the efficiency of operational units, particularly the infantry battalions of the infantry brigades under the Eastern and Western Commands, on delivery of readiness. However, a significant limitation in conducting such research is the secrecy of detailed information, especially regarding the output of the defense program, namely, the military capability and its indicators – readiness levels of units. Also, due to the peculiarities of the current defense program structure, it should be quite problematic to obtain accurate information regarding the inputs, for example, detailed costs per battalion for any particular period. Consequently, in order to demonstrate the possibility of using DEA for the above purposes, an illustrative analysis of the efficiency of battalions was carried out using fictitious data. Figure 1. Units, equipment, and personnel in an Army Infantry Brigade Combat Team, excluding support battalion [17] DSS Vol. 3, January 2022, pp.83-100 94 According to Strategic Defense Review (SDR) 2021-2025 [14], the GDF Future Force Structure, along with other military units, includes four infantry brigades under the Eastern and Western Commands. Due to the unavailability of relevant information regarding the structure of brigades of the GDF from open sources, data on the structure of brigades of the US Army were used as an example. Infantry Brigade Combat Teams (IBCTs) constitute the Army’s “light”, primarily foot-mobile ground forces that can move by foot, vehicle, or air (either air landed or by helicopter) [15]. According to the Army’s Field Manual (FM) 3-96, the IBCTs are employed as follows: “The role of the IBCT is to close with the enemy by means of fire and movement to destroy or capture enemy forces, or to repel enemy attacks by fire, close combat, and counterattack to control land areas, including populations and resources” [16, p. 1-2]. IBCTs are relatively independent tactical formations that are designed to include approximately 4,400 personnel [17]. As can be seen from Figure 1, there are three infantry battalions in the IBCT. Since there are four infantry brigades in the GDF Future Force Structure, it can be assumed that there will be a total of 12 infantry battalions that will be treated as decision making units (DMUs) in our sample DEA model. Two input variables can be defined for the use of personnel and equipment: (1) Personnel Costs - pay and benefits for military personnel, compensation for civilian employees, health care costs, and travel expenses for military and civilian personnel; and (2) Material Costs - daily expenses of operating a unit, such as equipment maintenance, training, support contractors, and so on. Readiness indicators by category can be used as outputs: personnel (P-level), equipment availability (S-level), equipment readiness (R-level) and training (T- level) (see Figure 2). Figure 2. Data model for the DEA It should be mentioned that according to Avkiran, “there are some rules of thumb on the number of inputs and outputs to select and their relation to the number of DMUs” [5, p. 115]. Boussofiane, Dyson and Thanassoulis argue that to obtain good discriminatory power out of the CCR and BCC models, the lower bound on the number of DMUs must be a multiple of the number of inputs and the number of outputs. For example, if there are 2 inputs and 4 outputs, the minimum total number of DMUs must be 8 for some discriminatory power to exist in the model [18]. DSS Vol. 3, January 2022, pp.83-100 95 Golany and Roll propose a rule of thumb that the number of units should be at least twice the number of inputs and outputs under consideration [19]. Bowlin mentions the need to have three times as many DMUs as there are input and output variables [20]. According to Dyson et al., a total of two times the product of the number of input and output variables is recommended [21]. For example, for a 3-input, 4-output model, Golany and Roll recommend 14 DMUs, while Bowlin recommends 21 DMUs. In any case, these figures should probably be used as the minimum for baseline performance models. As can be seen, the variants of DMUs, inputs and outputs proposed in our example basically meet the above requirements. Table 5 below shows the fictitious data for conducting the DEA. Table 5. Fictitious data for conducting the DEA DMU name Input 1 Personnel costs GEL Input 2 Material costs GEL Output 1 P-level (%) Output 2 S-level (%) Output 3 R-level (%) Output 4 T-level (%) 1st Infantry battalion 10 800 000 5 000 000 85 91 70 85 2nd Infantry battalion 10 200 000 5 500 000 75 82 75 85 3rd Infantry battalion 11 760 000 6 000 000 88 74 96 87 4th Infantry battalion 11 160 000 7 000 000 74 96 84 78 5th Infantry battalion 10 680 000 6 500 000 96 97 99 95 6th Infantry battalion 10 920 000 5 850 000 81 75 82 76 7th Infantry battalion 10 788 000 5 100 000 68 85 78 82 8th Infantry battalion 11 520 000 5 350 000 92 86 91 95 9th Infantry battalion 11 040 000 5 120 000 77 88 99 87 10th Infantry battalion 10 560 000 4 950 000 85 91 75 81 11th Infantry battalion 10 320 000 6 200 000 86 69 85 73 12th Infantry battalion 11 568 000 6 850 000 91 92 94 93 The exemplifying analysis was carried out using DEA-Solver-PRO 5.0 software [22]. As can be seen from the DEA test results depicted in Table 6 and 7, five of the twelve infantry battalions are efficient, while the rest show some inefficiency. Table 6. DEA test results Rank DMU Score 1 10th Infantry battalion 1 1 1st Infantry battalion 1 1 9th Infantry battalion 1 1 8th Infantry battalion 1 1 5th Infantry battalion 1 6 2nd Infantry battalion 0,970912603 7 7th Infantry battalion 0,954380827 8 4th Infantry battalion 0,947123379 9 3rd Infantry battalion 0,939071369 10 11th Infantry battalion 0,93254636 11 12th Infantry battalion 0,911386993 12 6th Infantry battalion 0,87500128 DSS Vol. 3, January 2022, pp.83-100 96 Table 7. Projections by the CCR Model No. DMU Score I/O Data Projection Difference % 1 1st Infantry battalion 1 Personnel costs GEL 10800000 10800000 0 0,00% Material costs GEL 5000000 5000000 0 0,00% P-level (%) 85 85 0 0,00% S-level (%) 91 91 0 0,00% R-level (%) 70 70 0 0,00% T-level (%) 85 85 0 0,00% 2 2nd Infantry battalion 0,970912603 Personnel costs GEL 10200000 9903308,547 -296691,4534 -2,91% Material costs GEL 5500000 5340019,314 -159980,6857 -2,91% P-level (%) 75 84,23988411 9,239884114 12,32% S-level (%) 82 82,23862868 0,238628682 0,29% R-level (%) 75 85,26924191 10,26924191 13,69% T-level (%) 85 85 0 0,00% 3 3rd Infantry battalion 0,939071369 Personnel costs GEL 11760000 11043479,3 -716520,7003 -6,09% Material costs GEL 6000000 5634428,214 -365571,7859 -6,09% P-level (%) 88 88 0 0,00% S-level (%) 74 90,00404513 16,00404513 21,63% R-level (%) 96 96 0 0,00% T-level (%) 87 92,00663008 5,006630078 5,75% 4 4th Infantry battalion 0,947123379 Personnel costs GEL 11160000 10569896,91 -590103,0928 -5,29% Material costs GEL 7000000 6432989,691 -567010,3093 -8,10% P-level (%) 74 95,01030928 21,01030928 28,39% S-level (%) 96 96 0 0,00% R-level (%) 84 97,97938144 13,97938144 16,64% T-level (%) 78 94,02061856 16,02061856 20,54% 5 5th Infantry battalion 1 Personnel costs GEL 10680000 10680000 0 0,00% Material costs GEL 6500000 6500000 0 0,00% P-level (%) 96 96 0 0,00% S-level (%) 97 97 0 0,00% R-level (%) 99 99 0 0,00% T-level (%) 95 95 0 0,00% 6 6th Infantry battalion 0,87500128 Personnel costs GEL 10920000 9555013,98 -1364986,02 -12,50% Material costs GEL 5850000 5118757,489 -731242,5106 -12,50% P-level (%) 81 81 0 0,00% S-level (%) 75 78,99387085 3,993870847 5,33% R-level (%) 82 82 0 0,00% T-level (%) 76 81,84403062 5,84403062 7,69% DSS Vol. 3, January 2022, pp.83-100 97 7 7th Infantry battalion 0,954380827 Personnel costs GEL 10788000 10295860,37 -492139,6349 -4,56% Material costs GEL 5100000 4867342,219 -232657,7807 -4,56% P-level (%) 68 78,80442417 10,80442417 15,89% S-level (%) 85 85 0 0,00% R-level (%) 78 78 0 0,00% T-level (%) 82 82 0 0,00% 8 8th Infantry battalion 1 Personnel costs GEL 11520000 11520000 0 0,00% Material costs GEL 5350000 5350000 0 0,00% P-level (%) 92 92 0 0,00% S-level (%) 86 86 0 0,00% R-level (%) 91 91 0 0,00% T-level (%) 95 95 0 0,00% 9 9th Infantry battalion 1 Personnel costs GEL 11040000 11040000 0 0,00% Material costs GEL 5120000 5120000 0 0,00% P-level (%) 77 77 0 0,00% S-level (%) 88 88 0 0,00% R-level (%) 99 99 0 0,00% T-level (%) 87 87 0 0,00% 10 10th Infantry battalion 1 Personnel costs GEL 10560000 10560000 0 0,00% Material costs GEL 4950000 4950000 0 0,00% P-level (%) 85 85 0 0,00% S-level (%) 91 91 0 0,00% R-level (%) 75 75 0 0,00% T-level (%) 81 81 0 0,00% 11 11th Infantry battalion 0,93254636 Personnel costs GEL 10320000 9623878,44 -696121,5601 -6,75% Material costs GEL 6200000 5781787,435 -418212,5652 -6,75% P-level (%) 86 86 0 0,00% S-level (%) 69 87,15708111 18,15708111 26,31% R-level (%) 85 88,04103148 3,041031479 3,58% T-level (%) 73 84,94507688 11,94507688 16,36% 12 12th Infantry battalion 0,911386993 Personnel costs GEL 11568000 10542924,74 -1025075,26 -8,86% Material costs GEL 6850000 6243000,905 -606999,0952 -8,86% P-level (%) 91 93,56101001 2,561010012 2,81% S-level (%) 92 93,80856701 1,808567007 1,97% R-level (%) 94 96,07991476 2,079914761 2,21% T-level (%) 93 93 0 0,00% Sherman [27] argues that the DEA is best used as a tool that can focus the attention of managers. In the infantry battalions example above, potential improvements suggest opportunities for managers (commanders) DSS Vol. 3, January 2022, pp.83-100 98 to explore in search of higher performance. The process includes identifying the main sources of inefficiency as well as those units that can become reference DMUs for others. 3. Conclusions The way to improve the efficiency and effectiveness of public spending, which is a top priority for any government in any country, implies the introduction of Performance-Based Budgeting (PBB). Program budgeting is one of the most advanced government-wide performance budgeting systems that systematically uses performance information in the preparation of the state budget where expenditures are classified into groups of similar activities or projects (i.e., programs) with common outputs and outcomes. It is important to keep in mind that without systematic development and use of program performance information and adequate and effective performance indicators, program budgeting in the defense sector does not make sense as a tool to improve the efficiency and effectiveness of the defense resource management process. Only by defining and tracking success can it be known if the defense organizations and units perform efficiently and effectively. In this article, DEA was considered an excellent mathematical programming and a powerful management tool that can be used to measure, evaluate, and analyze the efficiency of the state and government as a whole, as well as commercial and non-profit organizations, including military units. DEA can be applied as an instrument to hold managers at all levels accountable for their performance which is critical to effective Performance-Based Budgeting (PBB). In this study, Data Envelopment Analysis (DEA) has been applied to NATO members and some Eastern Europe post-Soviet aspirant and partner countries (Ukraine, Georgia and Moldova) to understand how efficient each country is at achieving its military power. The efficiency of the decision-making units was measured with a CCR model. Due to the secrecy and peculiarities of the current defense program structure, there are significant limitations in obtaining detailed information on the main output of the defense, namely, the military capability and its indicators – readiness levels of units and accurate cost information, such as detailed information on the cost per battalion for any given period. Therefore, in order to demonstrate the feasibility of using DEA to examine the efficiency of operational units, an illustrative analysis of the efficiency of the aforementioned infantry battalions was carried out using fictitious data. I believe that one of the main barriers to using DEA as a valid tool for measuring and evaluating the performance of the GDF units is the current structure of the defense program. The introduction of the defense program structure proposed in my recent article [23], which will include the Major Force Program developed on a Force Capabilities basis; the definition of capabilities embodied in a force element as the main output of the defense subprograms; and the identification of force (program) elements (e.g., departments, commands, brigades, and battalions) as cost centers, will facilitate the use of the DEA and other statistical tools to measure and evaluate the performance of the GDF units and develop proposals for its improvement. A key precondition for the successful application of the DEA and other statistical tools in measuring and evaluating performance in the Ministry of Defense is the development of effective computerized financial management information systems (FMIS), including computerized accounting systems. One of the challenges for the Ministry is to implement an effective management accounting system to provide managers (unit commanders) with timely, accurate, meaningful, and insightful information without which an effective decision-making process is impossible. Well-organized accounting systems are the main original source of the best quality and ultimately the most reliable primary data for the life cycle costing to support decision-making process as well as performance measurement and evaluation. It is also crucial to include the “readiness level” as an output indicator in the defense program structure of the Georgian Ministry of Defense. The target readiness levels of the GDF units should also be specified in the Defense Program Guidance (DPG) and other planning documents of the MOD of Georgia (in the secret part of DSS Vol. 3, January 2022, pp.83-100 99 the documents), as well as procurement objectives and descriptions of acceptable risk. Defense program managers who are accountable for the resources provided must monitor the balance of inputs to readiness and the state of readiness achieved. Disclaimer The views represented in this paper are those of the author and don’t reflect the official policy or position of the Ministry of Defense of Georgia. Declaration of competing interest The authors declare that they have no any known financial or non-financial competing interests in any material discussed in this paper. Funding information No funding was received from any financial organization to conduct this research. References [1] W. Van Dooren, G. Bouckaert, and J. Halligan, Performance Management in the Public Sector, Routledge, 2015, doi: 10.4324/9781315817590. [2] M. Robinson, Best practice in performance budgeting, Discussion paper No. 124, Queensland, 2002. https://tinyurl.com/5dh8eass [3] Northern Ireland Assembly, Methods of Budgeting, Research Paper 06/10, 2010. https://www.focusintl.com/RBM150-0610.pdf [4] Steering Committee for the Review of Commonwealth/State Service Provision, Data Envelopment Analysis: A technique for measuring the efficiency of government service delivery, Canberra, 1997. https://tinyurl.com/29wau6h4 [5] N. Avkiran, Productivity analysis in the service sector with data envelopment analysis, Queensland, 2006. http://dx.doi.org/10.2139/ssrn.2627576 [6] H. D. Sherman, Service Organization Productivity Management, the Society of Management Accountants of Canada, 1988. [7] M. J. Farrell, “The Measurement of Productive Efficiency,” Journal of the Royal Statistical Society. Series A (General), vol. 120, no. 3, pp. 253-290, 1957, doi: 10.2307/2343100. https://www.jstor.org/stable/2343100 [8] A. Charnes, W.W. Cooper, and E. Rhodes, “Evaluating Program and Managerial Efficiency: An Application of Data Envelopment Analysis to Program Follow Through,” Management Science, vol. 27, no. 6, pp. 668-697, 1981. https://www.jstor.org/stable/2631155 [9] Y. Ji and C. Lee, “Data envelopment analysis,” The Stata Journal, vol. 10, no. 2, pp. 267–280, 2010. https://www.stata-journal.com/article.html?article=st0193 [10] A. Charnes, W.W. Cooper, and E. Rhodes, “Measuring the efficiency of decision making units,” European Journal of Operational Research, vol. 2, pp. 429-444, 1978. https://tinyurl.com/2enk5vvh [11] Y. Ji and C. Lee, “Data envelopment analysis,” 267–280; Patricia Shewell and Stephen Migiro, “Data envelopment analysis in performance measurement: a critical analysis of the literature,” Problems and Perspectives in Management, vol. 14, no. 3-3, pp. 705-713, 2016. https://tinyurl.com/bdhv84uz [12] T. Hanson, Efficiency and productivity in the operational units of the armed forces, University of Oslo, Department of Economics, 2012. https://www.econstor.eu/handle/10419/90728. [13] T. Hanson, “Estimating output mix effectiveness: An applied scenario approach for the Armed Forces,” Omega: The International Journal of Management Science, vol. 83, pp. 39-49, 2019. https://tinyurl.com/yc7xurbu DSS Vol. 3, January 2022, pp.83-100 100 [14] Ministry of Defense of Georgia, Strategic Defense Review (SDR) 2021-2025, 2021. https://tinyurl.com/ya6bbe3p [15] Congressional Research Service, Infantry Brigade Combat Team (IBCT) Mobility, Reconnaissance, and Firepower Programs, 2019. https://crsreports.congress.gov [16] Department of the Army, FM 3-96 Brigade Combat Team, 2021. https://tinyurl.com/4tnejznm [17] Congressional Budget Office, The U.S. Military’s Force Structure: A Primer, 2021. www.cbo.gov/publication/57088 [18] A. Boussofiane, R.G. Dyson, and E. Thanassoulis, “Applied Data Envelopment Analysis,” European Journal of Operational Research, vol. 52, no. 1, pp. 1-15, 1991. https://tinyurl.com/yc768f95 [19] B. Golany and Y. Roll, “An Application Procedure for DEA,” International Journal of Management Science, vol. 17, no. 3, pp. 237-250, 1989. https://tinyurl.com/5dpafsny [20] W. F. Bowlin, “Measuring Performance: An Introduction to Data Envelopment Analysis (DEA),” Journal of Cost Analysis, vol. 15, no. 2, pp. 3-27, 1998. https://tinyurl.com/6xnuardk [21] R.G. Dyson, R. Allen, A.S. Camanho, V.V. Podinovski, C.S. Sarrico, and E.A. Shale, “Pitfalls and Protocols in DEA,” European Journal of Operational Research, vol. 132, no. 2, pp. 245-259, 2001. https://tinyurl.com/5cjp8a25 [22] W.W. Cooper, L.M. Seiford and K. Tone, User’s Guide to DEA-Solver-PRO (Professional Version 5.0), Springer, 2005. [23] I. Okromtchedlishvili, “PROPOSALS ON DEFENSE PROGRAM STRUCTURE: THE CASE OF THE MINISTRY OF DEFENSE OF GEORGIA,” Journal of Defense Resources Management (JoDRM), vol. 13, no. 1, pp. 88-105, 2022. http://www.jodrm.eu/issues/Volume13_issue1/06_OKROMTCHEDLISHVILI.pdf [24] N. Webb and D. Angelis, “Improving Performance Measurement in Defense Organizations,” The Journal of the American Society of Military Comptrollers, pp. 16-21, 2009. https://tinyurl.com/4ufr9rzr [25] J. Techau, The politics of 2 percent, NATO and the security vacuum in Europe, Carnegie Endowment for International Peace, 2015. https://tinyurl.com/5c5z6w8h [26] G. Wright, Program Budgeting Manual for Government of Georgia, 2015. https://tinyurl.com/2p8synhv [27] H.D. Sherman, “Data Envelopment Analysis as a New Managerial Audit Methodology-Test and Evaluation,” Auditing: A Journal of Practice and Theory, 4(1), pp. 35-53, 1984. https://tinyurl.com/4e9vfvrf [28] International Institute for Strategic Studies, The Military Balance 2021, Routledge, 2021. https://tinyurl.com/2y3nz8ck [29] https://data.worldbank.org/ [30] https://www.globalfirepower.com/ [31] https://www.nato.int/cps/en/natohq/news_184844.htm