A UNIVERSAL MATHEMATICAL LAW CRITERION FOR ALGORITHMIC VALIDITY Developments in Business Simulation and Experiential Learning, Volume 29, 2002 A UNIVERSAL MATHEMATICAL LAW CRITERION FOR ALGORITHMIC VALIDITY Dickinson, John R. University of Windsor bjd@uwindsor.ca ABSTRACT That simulation game algorithms behave validly is an obviously essential property underlying the effectiveness of simulations. While algorithmic validation does have a philosophical tradition, relatively few empirical validation studies have actually been undertaken. This study positions a mathematical law based in digital analysis as a viable criterion for simulation game algorithm validation and, by application, establishes the validity of a popular marketing management simulation game against that criterion. INTRODUCTION AND PURPOSE The use of simulation games for education and training is longstanding and widespread. The introduction of gaming dates from military strategy board games of 5,000 years ago (Wolfe & Crookall, 1998), though the Top Management Decision Simulation game developed in 1956 by the American Management Association is generally regarded as the first of the modern generation of business simulation games. It is estimated that 97 percent of universities accredited by the American Assembly of Collegiate Schools of Business (AACSB) use simulation games in their curricula and that 62 percent of U.S. corporations use business simulation games in their training programs (Faria, 1998). Too, simulation games have potential use as platforms for basic research, though this use has been limited and sporadic (Babbs, Leslie, & Van Slykes, 1966; Dickinson, 2001; Gentry et al., 1984). Simulation games have been the object of considerable research. Much of that research has examined the administration of games and their effectiveness as learning vehicles. Relatively little research, though, has attended the algorithms which lie at the heart of computerized simulation games and very little research has focused on the validity of those algorithms. The purpose of this study is to adapt a well- established criterion to the validation of simulation game algorithms. BACKGROUND Competitive simulation games pit one company against competing companies. This is in contrast to games played “against the computer.” The former incorporate the dynamic of the strategies of competing managers in determining a given company’s performance in addition to the aspect of a company’s strategy effectiveness vis-a-vis the simulated structural environment. In either mode, for marketing management simulation games specifically, the basic determination made by software algorithms is demand. Numerous financial results following from demand or sales are presented in the familiar income statement and balance sheet. Nonfinancial results such as sales force morale, product quality, and consumer attributes such as brand loyalty may also be determined. Total enterprise games that include, for example, human resources management and production management, generally make a variety of additional determinations such as employee turnover and mixes of resource requirements and costs. For marketing management games, the determination of demand might employ standard marketing models such as those found in Lilien & Kotler (1983). Additional comprehensive demand models within the simulation game context have been developed by Carvalho (1995), Goosen & Kusel (1993), Teach (1990), Thavikulwat (1989), and Gold & Pray (1983, 1984), among others. Several related algorithms for modeling specific aspects of enterprise management or operation have also been put forth. These include modeling economic environments (Arellano and Hopkins, 1992), consumer environments (Gold & Pray, 1997), cross-elasticities (Teach & Schwartz, 2000), product quality (Thavikulwat, 1992), new product development (Gold & Pray, 1999), total quality management (Mergen & Pray, 1992), and costs (Gold, 1992; Gold & Pray, 1992; Goosen, 1991; Goosen, Foote, & Terry, 1994). Another related genre of algorithms addresses the modeling of qualitative events such as innovative product design (Summers, 2000), advertising creative strategy (Cannon, McGowan, & Yoon, 1994; Cannon, 1993), and technological innovation (Pray & Methe, 1991). Considering this cursory review, much research has focused on the development of business simulation gaming algorithms. Relatively little research, however, has evaluated the validity of such algorithms. The validity of all of the above types of algorithms is an obvious requirement for the effectiveness of simulation games for both pedagogical and basic research purposes. The broad validity of simulations continues to be 26 mailto:bjd@uwindsor.ca Developments in Business Simulation and Experiential Learning, Volume 29, 2002 addressed both philosophically and empirically. Of the former, Feinstein & Cannon (2001), Peters, Vissers, & Heijne (1998), and Stanislaw (1986) have presented general frameworks for simulation validation. Wolfe & Jackson (1989) elaborated on the need for investigating algorithmic validity specifically. With respect to simulation algorithms, Naylor et al. (1966, p. 40) early on posed the approach of historical verification, i.e., “...how well do the simulated values of the endogenous variables compare with known historical data, if historical data are available?” Despite considerable developmental research and despite validity being an obviously essential property, relatively little investigation has been done of algorithmic validity. Following the historical verification approach, Napier & House (1990) presented a framework for examining relationships among key financial ratios for simulated and actual companies. House, Parks, & Lindstrom (1994) compared the relationship between research and development and profitability in two simulation games with that relationship in 50 companies (and found the simulations wanting). In the same mode, House & Taylor (1992, p. 213) loosely compared financial performance measures of two simulation games with unspecified “real world studies” and “real world experience.” In contrast, Thavikulwat (2000, p. 132) validated his currency valuation algorithm against a normative model, rather than against “everyday-world reality.” The present study adapts digital analysis to provide a generally applicable criterion for evaluating the validity of simulation game outcomes. Note that with this criterion it is not specific algorithms per se that are evaluated, but the results generated by those algorithms. DIGITAL ANALYSIS “Digital analysis...[is]...the analysis of digit and number patterns with the objective of detecting abnormal recurrences of digits, digit combinations and specific numbers.” (Nigrini & Mittermaier, 1997, p. 52) One form of digital analysis is based on Benford’s Law of Anomalous Numbers (Benford, 1938). Benford’s beguiling Law asserts that the first digits of numbers generated by no known systematic process do not occur with equal probability. Rather, such first digits follow a logarithmic pattern with 1s appearing most frequently through to 9s appearing least frequently. Specifically, Benford’s Law is: Probability (1st digit =n) = log (1+1/n) The Law is popularly attributed to Frank Benford. In his position as physicist with the General Electric Research Laboratories, Benford noticed that the early pages of logarithmic tables were more worn than later pages, indicating that logs for numbers beginning with 1 were sought more frequently than were logs for numbers beginning with 9. This led Benford to develop formulae not only for the expected frequencies of first digits, but also second and subsequent digits and digit combinations. Raimi (1976) compiled a comprehensive review of “the first digit problem,” including the observation (p. 522) that Newcomb (1881) had identified the phenomenon 57 years prior to Benford. Benford himself demonstrated that the Law holds for data from a wide variety of sources, though not all sources. Most recently, Huxley (1999) demonstrated that the first digits of 19,608 checks followed Benford’s Law “almost perfectly.” Benford’s Law has been applied to demonstrate that company reports of income inordinately just exceed key targets of 10k where k is a positive integer (Carslaw, 1988) and that net income and earnings per share figures tend to be rounded up (Thomas, 1989). A different type of “key target” is common in income tax tables, where income categories are defined and tax rates assigned to respective categories. Taxpayers have an incentive for their reported taxable income to fall below a defined category boundary. Christian & Gupta (1993) found such a bias toward incomes falling in the upper ranges of the categories and invoked Benford’s Law for third, fourth, and fifth digits to establish expected, i.e., non-tax- evasive, digit distributions. Nigrini & Mittermaier (1997) proposed using several extensions of Benford’s Law (second digits and first-two digits, among others) to select audit samples, i.e., samples of possibly fraudulent tax filings. In a production context, Becker (1982) described the use of Benford’s Law to identify systematic errors in mean-time-to- failure data. The numerous successful applications of Benford’s Law provide empirical support for the validity of the Law itself. Discussions attending these and other researches invariably consider why Benford’s Law is true, not if it is true. Regarding the distribution of the first digit, Benford’s Law can be mathematically derived (Feller, 1971, pp. 63-64; Pinkham, 1961). That is, at least regarding the first digit, Benford’s Law has been proven to be true. In sum, considerable evidence has accumulated that Benford’s Law is tenable for financial and related data. Benford’s Law thus seems appropriate as a criterion for the validity of the outcomes of business simulation game algorithms. More broadly, Benford’s Law should be applicable to simulation game algorithms generally. Benford’s original study (1938) found the Law to apply to data sets as diverse as numbers throughout articles appearing in periodicals such as newspapers and the Reader’s Digest, street addresses, and drainage areas of rivers. Thus, Benford’s Law is potentially a validation criterion for the gamut of simulation game algorithms. METHODOLOGY To illustrate the application of Benford’s Law to simulation game validity, financial performance data were gathered from a simulation competition using The Marketing Management Experience (MME) (Dickinson, 2000). In the MME, participants assume the role of marketing manager for a manufacturer of both still and video digital cameras. The 27 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 cameras may be marketed in one or both of two regions, the regions exhibiting different economic, demographic, and lifestyle profiles. The two regions and two products yield four region-product segments. Demand in each segment is fundamentally determined by 16 elements of a company’s marketing mix as formulated by simulation participants. Algorithms incorporate not only the “main effect” of each mix element on demand, but also lagged and cumulative effects and interactions. Too, the effectiveness of each company’s strategy is partly determined by the strategies of competing companies, e.g., a camera price of $600 has a different effect on demand where competitors’ prices are generally lower than $600 than when their prices are generally higher than $600. Participants in the research were 48 Masters of Business Administration students enrolled in an introductory marketing management course with each student managing his or her own simulation company. The 48 companies were grouped into eight industries of five companies each plus two industries of four companies each. Simulation managers formulated their marketing mixes for a total of nine periods plus an initial trial period. The MME algorithms, among other outcomes, yield a standard income statement. Two analyses are performed in this study. The first analysis, in keeping with published financial analysis applications (Carslaw, 1988; Nigrini & Mittemaier, 1997; Thomas, 1989), examines seven key income statement summary lines: gross sales, cost of goods sold, gross margin, total operating expenses, operating income, taxable income, and after tax earnings. The second analysis is more comprehensive. The MME income statement contains 24 lines. With the following exceptions, 19 lines (including the seven in the first analysis) were included in the second analysis. Three lines are fixed costs. A fourth item, sales promotion, has a menu of set costs and a fifth item, consulting fee, is levied at the discretion of the competition administrator. These items are “assigned” numbers not subject to Benford’s Law (Nigrini & Mittermaier, 1997, p. 55) and are excluded from the analyses. The MME actually produces five income statements: one for each of the four region-product segments and a fifth for the total company. Where required, common costs are allocated to segments on the basis of unit sales. It is the entries in these four segment income statements plus the aggregated total company income statement that comprise the data for this application of Benford’s Law. Analyses compare the percentages of first digits and the percentages of second digits with the respective percentages expected under Benford’s Law. RESULTS Results for the first digits of selected income statement summary lines are presented in Table 1. Across the 48 companies, nine periods of competition, five income statements per company, and four items per income statement, 14,791 nonzero first digits were available. (Benford’s seminal empirical tests were based on sample sizes of 2,968 to 20,229 observations.) For all observations the mean absolute deviation (MAD) totaled across all nine digits is 6.50 percentage points, or 0.72 percentage points per digit. However, for only four digits does the MAD exceed 0.50: 1s and 9s are under-represented, while 3s and 4s are over- represented. TABLE 1: First Digit, Seven Summary Lines from Income Statement First Digit Benford’s Law (%) Periods 1-9 a Periods 1-3 a Periods 4-6 a Periods 7-9 a 1 30.10 1.96 3.19 2.52 0.16 2 17.61 0.12 0.99 0.01 0.63 3 12.49 1.95 2.70 2.34 0.81 4 9.69 0.86 0.59 0.69 1.31 5 7.92 0.32 1.15 0.18 0.02 6 6.69 0.04 0.22 0.42 0.31 7 5.80 0.40 0.44 0.12 0.64 8 5.12 0.22 0.60 0.20 0.15 9 4.58 0.63 0.97 0.41 0.51 Total Deviation 6.50 10.85 6.90 4.54 Mean Devia- tion Per Digit 0.72 1.21 0.77 0.50 Observations 14,791 4,942 4,949 4,900 a Entries are mean absolute deviation across gross revenue, cost of goods sold, gross margin, total operating expenses, operating income, taxable income, and after tax earnings. 28 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Results for each third of the nine-period competition are also presented in Table 1. Mean absolute deviations per digit are somewhat larger, ranging from 0.77 to 1.21 percentage points. Deviations aggregated across all digits decrease as the competition progresses. However, for individual digits there are several exceptions and no pattern at the that level is apparent across the different phases of the competition. Results for the first digits of the more comprehensive set of 19 income statement items are presented in Table 2. With the larger set of income statement items, 34,568 nonzero first digits were available. For all observations the mean absolute deviation totaled across all nine digits was 2.86 percentage points, or 0.32 percentage points per digit. Across the nine digits, only a single deviation exceeded half a percentage point (3s deviated from the Law by 1.01 percentage points) and only two exceeded four-tenths of a percentage point. For each of the three three-period phases, mean absolute deviations per digit are only slightly larger, ranging from .40 to .53 percentage points. Again, no pattern is apparent across the different phases of the competition . TABLE 2: First Digit, Nineteen Itemized Lines from Income Statement First Digit Benford’s Law (%) Periods 1-9 a Periods 1-3 a Periods 4-6 a Periods 7-9 a 1 30.10 0.08 1.02 0.19 1.08 2 17.61 0.42 0.41 0.85 0.81 3 12.49 1.01 1.74 1.02 0.25 4 9.69 0.36 0.50 0.58 0.00 5 7.92 0.24 0.14 0.23 0.36 6 6.69 0.08 0.28 0.26 0.24 7 5.80 0.02 0.05 0.10 0.14 8 5.12 0.34 0.25 0.24 0.55 9 4.58 0.30 0.41 0.16 0.33 Total Deviation 2.86 4.79 3.63 3.75 Mean Devia- tion Per Digit 0.32 0.53 0.40 0.42 Observations 34,568 11,546 11,807 11,215 a Entries are mean absolute deviation across 19 income statement lines, e.g., advertising expense. The mean absolute deviations per digit of 0.72 and 0.32 for the respective analyses compare with Nigrini & Mittermaier’s MAD of 0.44 (1997, p. 59) and lead to the same conclusion that the first digits conform to Benford’s Law. The MME MADs are well within the 0.96 and 1.04 deviations that Carslaw (1988) found for companies’ reported incomes and which he deemed to “not deviate widely” and to conform to expectations (p. 323). Distributions of the second digits of the same sets of data, i.e., seven summary lines and 19 itemized lines from the income statement, respectively, were also compared with Benford’s Law. The distributions of second digits follow Benford’s Law even more closely than those of the first digits. Across the seven summary lines and all 10 digits, the mean absolute deviation per digit was .27 percentage points (Table 3). Only one of the 10 MADs exceeds .40 of a percentage point. The respective MADs for each third of the nine-period competition are .35, .37, and .49 of a percentage point, again, very close fits with Benford’s Law. 29 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 TABLE 3: Second Digit, Seven Summary Lines from Income Statement Second Digit Benford’s Law (%) Periods 1-9 a Periods 1-3 a Periods 4-6 a Periods 7-9 a 0 11.97 0.09 0.27 0.09 0.07 1 11.39 0.24 0.23 0.13 1.08 2 10.88 0.09 0.46 0.60 0.14 3 10.43 0.89 0.78 0.75 1.15 4 10.03 0.38 0.37 0.35 0.42 5 9.67 0.38 0.49 0.11 0.76 6 9.34 0.25 0.21 0.71 0.25 7 9.04 0.07 0.01 0.39 0.58 8 8.76 0.20 0.32 0.53 0.24 9 8.50 0.09 0.40 0.09 0.23 Total Deviation 2.68 3.54 3.74 4.93 Mean Devia- tion Per Digit 0.27 0.35 0.37 0.49 Observations 14,791 4,942 4,949 4,900 a Entries are mean absolute deviation across gross revenue, cost of goods sold, gross margin, total operating expenses, operating income, taxable income, and after tax earnings. DIAGNOSTIC PURPOSE Results for the larger set of second digits of 19 income statement items illustrate the diagnostic purpose to which Benford’s Law may be applied in addition to its algorithmic validation purpose. The mean absolute deviation per digit is .54 (Table 4). That in itself suggests that the MME algorithms yield valid distributions of second digits. However, zeros are over-represented by 1.66 percentage points, a figure approaching twice the next highest absolute deviation (5s being over-represented by .89 percentage points). Several MME strategy decisions are restricted to thousand or hundred dollar units, for example, research and development expenditures. Thus, expenditures of less than $11,000 must necessarily have a zero as the second digit. Costs of adding, discontinuing, and maintaining discount stores are $4,000, $2,000, and $500 per store, respectively, again leading to numerous income statement entries having zero as the second digit. Each of the 34,561 second-digit observations was examined to determine the percent of values having zeros for all digits but the first. The income statement items having the highest incidences were, indeed, research and development (11.8%) and discount stores (11.6%), confirming the above explanation for the relatively greater deviation from Benford’s Law of zeros compared to other values. Had such explanations not been available, then attention might be directed to the algorithms determining these values as possibly being unrealistic. Reasons why the preponderance of zeros is not even greater are ready. Where total research and development expenditure is greater than $10,000 then the second digit is not constrained to be zero, with a similar rationale applying to discount store expenditures and other income statement items. Also, as described earlier, individual segment income statements are analyzed in this study in addition to the total company income statement. Several items in the segment income statements are prorations of the total company items, again removing the imposition of a zero as the second digit. Benford’s Law may serve a useful diagnostic purpose in identifying exceptional results which, in turn, must be explainable for the validity of the simulation algorithms to remain in tact. 30 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 TABLE 4: Second Digit, Nineteen Itemized Lines from Income Statement Second Digit Benford=s Law (%) Periods 1-9 a Periods 1-3 a Periods -6 a Periods 7-9 a 0 11.97 1.66 1.63 1.58 1.77 1 11.39 0.67 0.60 0.39 1.03 2 10.88 0.16 0.07 0.17 0.38 3 10.43 0.68 0.66 0.89 0.48 4 10.03 0.20 0.02 0.19 0.81 5 9.67 0.89 0.66 0.87 1.15 6 9.34 0.17 0.21 0.19 0.12 7 9.04 0.20 0.41 0.36 0.18 8 8.76 0.33 0.24 0.45 0.29 9 8.50 0.45 0.09 0.54 0.74 Total Deviation 5.42 4.59 5.64 6.97 Mean Devia- tion Per Digit 0.54 0.46 0.56 0.70 Observations 34,561 11,542 11,806 11,213 a Entries are mean absolute deviation across 19 income statement lines, e.g., advertising expense. CONCLUSION The validity of simulation gaming is sine qua non for the genre, with concern for validity attending many aspects of gaming. The numeric results of games is one of those aspects. Benford’s Law is a well-established mathematical law and is an ideally and seemingly universally suited criterion for the validity of simulation game numeric outcomes. This study has introduced Benford’s Law for this purpose and has provided an example of a successful application. Employing Benford’s Law to validate simulation game outcomes has several implications. First, validated games enhance the credibility of simulation gaming as a pedagogical and basic research medium. Second, software developers might use Benford’s Law toward ensuring that their products are valid. Identifying specific outcomes that should be expected to follow Benford’s Law but do not can serve a diagnostic purpose. Third, prospective users of a simulation game might expect assurance that the outcomes of the game comply with Benford’s Law, i.e., they comprise a Benford Set. REFERENCES Arellano, Fernando E. & Hopkins, Willie E. (1992). Modelling economic environments in business simulations: some comparisons and recommendations. In Gosenpud, Jerry & Gold, Steven (Eds.), Developments in Business Simulation & Experiential Exercises, Volume 19 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 7-10. Babb, E. M., Leslie, M. A., & Van Slyke, M. D. (1966). The potential of business-gaming methods in research. The Journal of Business, XXXIX(October), 465-472. Benford, F. (1938). The law of anomalous numbers. Proceedings, American Philosophical Society, 78, 551- 572. Becker, P. (1982). Patterns in listing of failure-rate and MTFF values and listings of other data. IEEE Transactions on Reliability R-31, 2(June), 132-134. Cannon, Hugh M. (1993). Incorporating advertising creative strategy into computer-based business simulations. In Gold, Steven C. & Thavikulwat, Precha (Eds.), Developments in Business Simulation & Experiential Exercises, Volume 20 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 120. Cannon, Hugh M., McGowan, Laura, & Yoon, Sung-Joon (1994). Incorporating advertising strategy into computer-based business simulations: a validation study. In Thavikulwat, Precha & Overby, John D. (Eds.), Developments in Business Simulation & Experiential Learning, Vol. 21 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 57-62. Carslaw, Charles A. P. N. (1988). Anomolies [sic] in income numbers: evidence of goal oriented behavior. The Accounting Review, LXIII(April), 321-327. 31 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Carvalho, Gerard F. (1995). Modeling the law of demand in business simulators. Simulation & Gaming, 26(March), 60-79. Christian, C. & Gupta, S. (1993). New evidence on >secondary evasion=. The Journal of the American Taxation Association (Spring), 72-92. Dickinson, John R. (2001). A need to revamp textbook presentations of price elasticity. In Krishnan, Ram & Viswanathan, Madhu (Eds.), Proceedings, American Marketing Association Winter Educators= Conference, Volume 12 (Chicago: American Marketing Association), 223-224. Dickinson, John R. (2000), The marketing management experience. Windsor, Ontario: Management Experiences. Faria, A. J. (1998). Business simulation games: current usage levelsBan update. Simulation & Gaming, 29(September), 295-308. Feinstein, Andrew Hale & Cannon, Hugh M. (2001). Fidelity, verifiability, and validity of simulation: constructs for evaluation. In Pittenger, Khushwant and Vaught, Mary Jo (Eds.), Developments in Business Simulation and Experiential Learning, Volume 28 (Association for Business Simulation and Experiential Learning), 57-67. Feller, William (1971). An introduction to probability theory and its applications, Volume II, Second Edition. New York: John Wiley & Sons, Inc. Gentry, James W., Tice, Thomas F., Robertson, Charles J., & Gentry, Martha J. (1984). Simulation gaming as a means of researching substantive issues: another look. In Currie, David M. & Gentry, James W. (Eds.), Developments in Business Simulation & Experiential Exercises, Volume 11 (Association for Business Simulation and Experiential Learning), 1-5. Gold, Steven C. (1992). Modeling short-run cost and production functions in computerized business simulations. Simulation & Gaming, 23(December), 417- 430. Gold, Steven C. & Pray, Thomas F. (1999). Changing customer preferences and product characteristics in the design of demand functions. Simulation & Gaming, 30(September), 264-282. Gold, Steven C. & Pray, Thomas F. (1997). Modeling attributes in demand functions of computerized business simulations: an extension of Teach=s gravity flow algorithm. In Butler, John K., Jr. & Leonard, Nancy H. (Eds.), Developments in Business Simulation and Experiential Learning, Volume 24 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 132-141. Gold, Steven C. & Pray, Thomas F. (1992). Modeling short- run cost and production functions in computerized business simulations. Simulation & Games, 23(4), 417- 430. Gold, Steven C. & Pray, Thomas F. (1984). Modeling market- and firm-level demand functions in computerized business simulations. Simulation & Games, 15(3), 346-363. Gold, Steven C. & Pray, Thomas F. (1983). Simulating market and firm level demandBa robust demand system. In Graf, L. A. & Gentry, J. W. (Eds.), Developments in Business Simulation and Experiential Exercises, Volume 10 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 240-243. Goosen, Kenneth R. (1991). Increasing simulation realism through the modeling of step costs. In Wheatley, Walt & Gosenpud, Jerry (Eds.), Developments in Business Simulation & Experiential Exercises, Volume 18 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 38-43. Goosen, Kenneth R., Foote, Paul, & Terry, Andy (1994). Increasing the effectiveness of performance evaluation through the design and development of realistic finance algorithms. In Thavikulwat, Precha & Overby, John D. (Eds.), Developments in Business Simulation & Experiential Learning, Volume 21 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 63-67. Goosen, Kenneth R. & Kusel, Jimie (1993). An interpolation approach to developing mathematical functions for business simulations. Simulation & Gaming, 24(March), 76-89. House, William C., Parks, Don M. & Lindstrom, Grant L. (1994). Relationships between R&D and Profitability: An Exploratory Comparison of Two Business Simulations with Two Real-World, Technology Intensive Industries. In Thavikulwat, Precha & Overby, John D. (Eds.), Developments in Business Simulation & Experiential Learning, Volume 21 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 75-78. House, William C. & Taylor, Lewis A. (1992). The lagged effects of decision variables on financial performance measures used in two business simulations. In Gosenpud, Jerry & Gold, Steven (Eds.), Developments in Business Simulation & Experiential Exercises, Volume 19 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 213. Huxley, Stephen J. (1999). Why Benford’s law works and how to do digit analysis on spreadsheets. International Conference of the Decision Sciences Institute, July 1999. Lilien, Gary L. & Kotler, Philip (1983). Marketing decision making: a model-building approach. New York: Harper & Row, Publishers. Mergen, A. Erhan & Pray, Thomas F. (1992). Modeling total quality elements into a strategy-oriented simulation. Simulation & Gaming, 23(September), 277-297. Napier, Herman S. & House, William C. (1990). Simulated versus actual firms: a ratio interaction matrix comparison. Simulation & Gaming, 21(June), 166-180. 32 Developments in Business Simulation and Experiential Learning, Volume 29, 2002 Naylor, Thomas H., Balintfy, Joseph L., Burdick, Donald S., & Chu, Kong (1966). Computer simulation techniques. New York: John Wiley & Sons, Inc. Newcomb, Simon (1881). Note on the frequency of use of the different digits in natural numbers. American Journal of Mathematics, 4, 39-40. Nigrini, Mark J. & Mettermaier, Linda J. (1997). The use of Benford’s law as an aid in analytical procedures. Auditing: A Journal of Practice and Theory, 16(Fall), 52-67. Peters, Vincent, Vissers, Geert, & Heijne, Gerton (1998). The validity of games. Simulation & Gaming, 29(March), 20-30. Pinkham, R. S. (1961). On the distribution of first significant digits. Annals of Mathematical Statistics, 32, 1223- 1230. Pray, Thomas F. & Methe, David T. (1991). Modeling radical changes in technology within strategy-oriented business simulations. Simulation & Gaming, 22(March), 19-35. Raimi, Ralph A. (1976). The first digit problem. The American Mathematical Monthly, 83 (August- September), 521-538. Stanislaw, Harold (1986). Tests of computer simulation validity. Simulation & Games, 17(June), 173-191. Summers, Gary (2000). Modeling innovation as a process of design in educational business simulations. In Morgan, Sandra & Page, Diana (Eds.), Developments in Business Simulation and Experiential Learning, Volume 26 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 146-152. Teach, Richard D. (1990). Demand equations for business simulations with market segments. Simulation & Gaming, 21(December), 423-442. Teach, Richard D. & Schwartz, Robert G. (2000). Introducing cross-elasticities in demand algorithms. In Page, Diana & Snyder, L. T. (Eds.), Developments in Business Simulation and Experiential Learning, Volume 27 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 125-131. Thavikulwat, Precha (2000). Validating a model of currency valuation. In Page, Diana & Snyder, L. T. (Eds.), Developments in Business Simulation and Experiential Learning, Volume 27 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 132- 137. Thavikulwat, Precha (1992). Product quality in computerized business simulations. Simulation & Gaming, 23(December), 431-441. Thavikulwat, Precha (1989). Modeling market demand in a demand-independent business simulation. Simulation & Games, 20(December), 439-458. Thomas, J. (1989). Unusual patterns in reported earnings. The Accounting Review (October), 773-787. Wolfe, Joseph & Crookall, David (1998). Developing a scientific knowledge of simulation/gaming. Simulation & Gaming, 29(March), 7-19. Wolfe, Joseph A. & Jackson, R. (1989). An investigation of the need for algorithmic validity. Simulation & Games, 20(September), 272-291. 33 Table of Contents Volume 29, 2001 Threshold Marketer A Family Of Marketing Simulations: Basic Marketer And Advanced Marketer Team Mode And Solo Mode Globalization As An Extended Experiential Exercise The Benefits And Planning Considerations Of Short Term Study Abroad Programs How 2 Setup Your Office Computer To Run Linux For Teaching E-Commerce Without Messing Everything Else Up Incorporating Cosmopolitan-Related Focus-Group Research Into Global Advertising Simulations Demonstration Of Advanced Features In Computer-Assisted Gaming Of International Business A Comparison Of Discrimination-Based Versus Conventional Simulation Game Scoring A Universal Mathematical Law Criterion For Algorithmic Validity The Impact Of Public Policy On Innovation: A Simulation Project For Research And Teaching Participant Identification Of Competitors In A Marketing Simulation Competition Simulation Research In The Hospitality Industry "Computer Simulation, Games And Roleplay: Drawing Lines Of Demarcation" Simulation Distribution Alternatives: Author/User Considerations Managing The Curiosity Gap Does Matter: What Do We Need To Do About It? Use Of External Interventions In A Computer Based Simulation Putting Service Learning Into Orbit It's A Wonderful Life: Simulating The Golden Years Adventures In Creating An Outdoor Leadership Challenge Course For An Emba Program Vbotz: A Pedagogical Cross-Disciplinary, Multi-Academic Level Manufacturing Corporate Simulation Use Of Computer Modeling In Management Accounting Is Simulation Performance Related To Application? An Exploratory Study Learning Cooperatively May Not Be Learning Collaborately! Perception Is Reality: Sharing Frames International Management Virtual Teamwork: A Simulation Financial Plan For Your Life And Career Goals Using Project-Based Experiential Learning Groups In The Principles Of Marketing Course Futures Course: Learning How To Anticipate The Future Of Business Interactive Online Strategic Market Planning With The Web-Based Boston Consulting Group (BCG) Matrix Graphics Package Strategy Learning In A Total Enterprise Simulation Investigation Of The Impact Of Decision Parameters For A Dutch Auction Simulation For Ipo Issues Integrating In-Class Learning With Out-Of-Classroom Experiences Through A Managerial Competency Development Framework War And Peace: Managing Students Learning Experience In A Competitive Simulation Game Virtually Experiential Classrooms Exercise: Conducting Role Plays Using Student Generated Cases Procedural Justice And Acceptance In Group Decision Making The E-Commerce Game: A Strategic Business Board Game Does Student Preparation Matter In A Simulation? A Comparison Of Pedagogical Styles The Game Of Business - A Weekend MBA Course Volume-Dependent Money Exchange Model For Gaming Simulations Implementing Service Learning For Accountants: The Not For Profit Project To Teach Vikings To Behave Among Mandarins: Lessons From Teaching With A Simulation Model Of Applied Business Ethics In International Management Maze Bright Teachers In The Classroom The Validity Investigation Of A Test Assessing Total Enterprise Simulation Learning What Makes Strategy Possible: An Illustration Using Paper And Scissors Learning Micro-OB Skills While Making Top Management Decisions In A Multinational Industrial Firm The Absel Research Heritage And The Bkl: Leveraging Their Value For Future Research New Product Development (Npd) Simulations: Some Challenging Questions And Tough Modeling Issues The Biofeedback Stress Test The Power Circle Exercise Total Enterprise Simulation Learning Compared To Traditional Learning In The Business Policy Course A Business Game Distance Education Application: Learning Outcomes And Experiences Is the Tobin's Q a good Indicator of a Company's Performance? A Critical Examination of the "Experiential" Premise Underlying BUsines Simulation Usage