IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 502 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 AHP STRUCTURING IN BEST-WORST SCALING AND THE SECRETARY PROBLEM Stan Lipovetsky Senior Research Director, GfK North America Marketing Science, USA Stan.lipovetsky@grk.com ABSTRACT The Analytic Hierarchy Process (AHP) is the widely known method and methodology of multiple criteria decision making, which enriches many other areas of mathematical and statistical data analysis. This work considers an extension of AHP hierarchical structuring by incorporating it into another method of prioritization known in marketing research as Best-Worst scaling (BWS). BWS is used for finding choice probabilities among the compared items, but when there are a large number of items it is rather difficult to apply this approach directly to all the items. The AHP methodology of hierarchical structuring and estimation of local priorities that are then synthesized into global preferences permits one to build BWS nested models to facilitate choice evaluations. For instance, the compared items can be divided into several subsets by the criteria of brand, size, packaging, etc. The BWS balanced designs and data eliciting procedure can be applied to each of these groups separately, with additional comparisons among the criteria. Synthesizing local choice probabilities by the priorities of the criteria yields global probabilities for the items of choice. In this paper we also apply another simple approach, the so-called “secretary problem” from the operations research field, for comparison. Numerical results demonstrate that these techniques can be very useful for prioritization problems in marketing research where there are a large number of items. Keywords: AHP; BWS; hierarchical structuring; secretary problem; choice probability 1. Introduction The Analytic Hierarchy Process (AHP) is the widely known method for multiple criteria decision making introduced by Thomas Saaty (1980), which has been developed and applied in numerous works (Saaty 1994, 1996, 2000; Golden et al., 1989; Saaty and Peniwati, 2012; Lipovetsky, 1996, 2009, 2010, 2011, 2013). The relation of AHP to other methods of decision theory has been studied in various works. Comparing AHP with multi-attribute utility theory (MAUT) can be found in Saaty (1980), Belton (1986), Vargas (1987), Winkler (1990) and Gass (2005). Practical utility estimations are usually performed in discrete choice modeling (DCM) for finding utility parameters and choice probabilities via multinomial-logit (MNL) models (McFadden, 1973, 1981; McFadden and Richter, 1990; Train, 2003). The comparison between AHP and the DCM conjoint models used in marketing research have been carried out in various studies (Mulye, 1998; mailto:Stan.lipovetsky@grk.com IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 503 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Scholl et al., 2005; Meißner et al., 2008, 2010; Scholz et al., 2007, 2010; Kallas et al., 2007, 2011; Ijzerman et al., 2012). The main conclusion of those works is that although the methods differ the resulting preference structures are pretty similar on the aggregate level, and on the individual level the AHP often demonstrates higher accuracy in choice prediction. The current work notes the possibility of extending the hierarchical structure used in AHP to another method for prioritizing items, namely, the Best-Worst Scaling (BWS), also known as the Maximum Difference (MaxDiff) approach. BWS is a contemporary technique widely used in marketing research for estimating choice probabilities for many items. It was proposed by Jordan Louviere (1991, 1993), and advanced in numerous works (for instance, Louviere et al., 2000; Marley and Louviere, 2005; Sawtooth 2007; Orme, 2010). In BWS, each respondent is presented with several subsets of a few items by way of a balanced design. A respondent answers which of the presented items is the best and which is the worst, and individual utilities are estimated in BWS using multinomial logit (MNL) modeling. Averaging the individual choice probabilities that were found yields the aggregated probabilities. The latter ones can be evaluated by the analytical formulae in the closed-form solution (Lipovetsky & Conklin, 2014a,b). Hierarchically structured BWS is similar to the nested logit of the generalized extreme value (GEV) models which have a rich elasticity structure in comparison with regular DCM (Ben-Akiva & Lerman, 1985; Wen & Koppelman, 2001; Hensher et al., 2005). The hierarchical approach to the choice based-conjoint (CBC) models widely used in marketing research corresponds to the adaptive choice based-conjoint (ACBC) models used for eliciting sequential data by comparing the best choices from the previous subsets of the items (Orme, 2006; Chrzan & Yardley, 2009; Netzer & Srinivasan, 2011; Wirth & Wolfrath, 2012; Sawtooth Software, 2014). In contrast to GEV and ACBC, the suggested approach using AHP requires neither a complicated actual nesting of MNL models, nor a complex scheme of data eliciting, so it is free of their computational burdens. 2. AHP combined with BWS This work considers AHP combined with BWS, so let us call this technique AHP-BWS. The AHP is used not in its entire methodological approach but only in its hierarchical structuring of the items under comparison, composing the local preferences into the global ones by weighting them by the importance of the criteria. A hierarchical configuration can be outlined by scrutinizing the connections among the alternatives in order to combine them into groups of different criteria. Often the compared BWS items can be divided into several subsets, for instance, by the criteria of brand, size, packaging, etc. Then, the BWS balanced design plans and data eliciting procedure can be applied to each of those subsets separately, with an additional BWS comparison among the criteria themselves. Synthesizing the local subsets of priorities weighted by the criterion preferences yields the global choice probability for all the items. AHP-BWS structuring can be especially useful for data with many dozens of items, because dividing them into relatively small subgroups significantly facilitates the difficulties of balanced designs, data elicitation and estimation. Possible optimal parameters of the hierarchy can be IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 504 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 evaluated for assembling a structure with the minimum number of needed comparisons among the alternatives and criteria (Lipovetsky, 2006). There always could be research problems requiring BWS estimation for a very large number of items, say, a hundred items for prioritization evaluated by thousands of responses. For such situations another approach is useful – it is based on what is known in operations research as the “secretary problem”, also called the best choice problem, and related to the hiring and finite horizon algorithms (Chow et al., 1964; Presman & Sonin, 1972; Vanderbei, 1980; Rose, 1982; Freeman, 1983; Petruccelli, 1984; Samuels, 1985; Ferguson, 1989; Stein et al., 2003; Bearden, 2006). In this paper we will show that results by different techniques are close to each other, and produce similar rankings among the choices. This means that the simple techniques can be successfully implemented on large data sets to serve various practical aims of managerial decisions in marketing research. For an explicit example, let us use data from a real marketing research project with 3,062 respondents who evaluated seventeen products. Each respondent received ten tasks of a balanced design of four items shown together, of which they were to indicate which items are the best and the worst ones. Using responses only on the choice of the “Best” item, the data can be modeled in a DCM approach. As is well known in DCM applications, each task can be presented in several rows, by the number of the items shown in one task, and such an extended matrix of data is completed with the additional column of the binary outcome where the “best” of these items is indicated by the value 1, and other values equal zero (more detail in Lipovetsky and Conklin, 2014a,b). DCM corresponds to a choice among several outcomes and can be described by a multinomial-logit model with the probability of a choice presented as the following: ,,...,2,1, )exp( )exp( 1 mk xa xa P m j jj kk k    (1) where xj are the variables of binary values indicating it was a j-th item shown to an i-th respondent in a given set or not. The parameters ak are the so-called utilities defining the probability of each k-th choice among all m of them, and the Pk are the binary outcome values of the “best” choice. When ak in Equation 1 are estimated, the probabilities of different choices are found by this MNL model. Repeating the DCM design for the “Worst” item chosen, we also obtain four rows for each task, and the outcomes indicated with a value that marks which item is the worst choice. DCM problem of the worst choice item uses the same approach as in Equation 1. For a simultaneous estimation of all the best and worst choices in one combined data set the following property is applied: if we change the signs of all predictors then the choice probability defines the absence of a binary event. It means that the design variables in the matrix of the “worst” choices can be taken with a negative sign, keeping the outcome the same as it already is. Then in BWS, the “best” and “worst” matrices, together with the corresponding outcome variable of the choices, can be stacked by rows into one combined dataset. With such a design, the positive and negative values of the binary predictors will push the outcomes with the values of 1 to the sides of the maximum and IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 505 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 minimum choice probability, respectively. Then the HB-MNL model produces individual parameters for each respondent. Averaging the individual choice probabilities yields the total choice preferences of the alternatives under consideration. Finding utilities and choice probabilities can be performed using specialized software, (for instance, Sawtooth Software, 2007). As it is shown in Lipovetsky and Conklin (2014a the estimation of the choice probability on the aggregate level is possible in the Closed-Form Analytical Solution (let us denote it as CFANS). With counts of the “best” and “worst” choices best j N and worst j N , proportions of the best and worst choices in total j N are j worst j worst jj best j best j NNfNNf /,/  , and the hit rate defined by their difference 2/)1(2/)1( worst j best jjj fffp  . Then the choice probability (Equation 1) for each item is given by the CFANS formula:                          n j x j j x j j m j jj kk k jk f f f f xa xa P 1 1 1 1 / 1 1 )exp( )exp( . (2) where in an i-th row the value is xij=1 if the j-th item is presented and xij=0 if it is absent. Consider this problem using the AHP-BWS approach. Suppose the products belong to three brands, so they can be put into three buckets. As shown in Table 1, in this example there are 6, 7, and 4 items in the 1 st , 2 nd , and 3 rd bucket, respectively. Instead of using BWS applied to all 17 items, we gather BWS data for each bucket separately, plus one additional set of comparisons among the buckets themselves. This significantly reduces the number of tasks, that is, the combinations of items, presented to each respondent. Indeed, there are 2,380 combinations from 17 by 4, so it is impossible to present all of them to each respondent, and only a small portion of those combinations would be found in an optimum design to be shown to respondents. On the other hand, to implement a hierarchical structure with the items considered within each category, in the first bucket there are 15 combinations from 6 by 4, so it is even possible to present them all, or at least most of them, to each respondent. Also, in the second bucket there are 35 combinations from 7 by 4, so it is possible to present at least half of them to each respondent. And in the third bucket there is only 1 combination from 4 by 4, so it can be presented to each respondent. For the higher level bucket of comparisons between the three categories, all three alternatives can be shown to each respondent in only one task. Thus, the total number of tasks can be small, but at the same time most of combinations for choices can be shown and checked with each respondent. It makes all the estimation of utilities and choice probabilities more reliable in comparison with a regular non-structured BWS approach. IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 506 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Table 1 Choice probability within and between buckets, in bucket order Frequency in buckets Bucket AHP-BWS Bucket Items Best item Worst item Hit Rate choice preference item choice probability 1 1 0.224 0.173 0.525 0.255 0.043 1 4 0.425 0.146 0.640 0.255 0.054 1 10 0.257 0.097 0.580 0.255 0.048 1 11 0.042 0.715 0.164 0.255 0.012 1 12 0.365 0.098 0.633 0.255 0.054 1 16 0.099 0.500 0.300 0.255 0.023 2 2 0.050 0.220 0.415 0.411 0.058 2 3 0.762 0.005 0.879 0.411 0.158 2 5 0.316 0.165 0.576 0.411 0.087 2 6 0.035 0.613 0.211 0.411 0.027 2 7 0.281 0.151 0.565 0.411 0.085 2 9 0.252 0.131 0.560 0.411 0.084 2 14 0.052 0.585 0.234 0.411 0.030 3 8 0.027 0.440 0.294 0.334 0.031 3 13 0.807 0.029 0.889 0.334 0.118 3 15 0.083 0.335 0.374 0.334 0.040 3 17 0.082 0.196 0.443 0.334 0.049 Table 1 shows the choice proportions of the best and worst items, and the hit rate (described with Equation 2), or the absolute choice probability of each of the items, separately defined within each bucket. The next column in Table 1 shows the preferences between different buckets, and their product with the hit rate normalized by one is presented in the last column of the synthesized AHP-BWS choice probabilities. Besides BWS, the following estimation for the “secretary problem” (SP), of operations research is possible. Briefly, SP can be described as the optimum strategy to maximize the probability of finding the best item when a subset of them is presented to the decision maker. As is known, the best choice in SP is given by the stopping rule of n/e subsamples (n is total number of items, and e is the base of natural logarithm), and picking the next outperforming item. It can also be seen in terms of the Gumbel distribution    uxaxxF n  expexp)( , where xn is the maximum x from a sample of size n, a is the scaling parameter reciprocal to the standard deviation, and u is the mode of this distribution, so x-u is the error term. Each unobserved component of utility in the discrete choice model in defined by the Gumbel distribution, and the difference between two extreme value variables is the logistic distribution (McFadden, 1973; Train, 2003). The point of mode x = u Gumbel distribution yields    37.0/10expexp)(  euxF n . In practical terms, it means that for many items we need to consider only about n/e number of them. In the case of many items, we can present to each respondent only one subset of about 0.37*n items and ask which IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 507 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 items are the best and worst ones. Then, estimation for the utilities and choice probabilities can be performed using the BWS approach. For the example using the data described above, Table 2 presents the results of regular BWS in comparison with the new approaches. For all the items this table shows the analytical solution for choice probability, then the probability obtained by Sawtooth software and by its adjustment advised in BWS applications for smoothing (Sawtooth Software, 2007). In the final two columns, the results of the secretary problem and of AHP-BWS are shown. The results vary across the techniques but in general the order of prioritization remains almost the same, especially for the higher probabilities that are the most important items. Sawtooth results are skewed towards the best choices, but the adjustment makes them more evenly distributed, and the analytical estimations are between them. The results of the Secretary Problem (SP) and AHP-BWS are close to each other and to the Sawtooth Adjusted results. Table 2 gives the choice probability estimates. Table 3 presents the regular pair correlations between the vectors of priorities, that is, the columns of Table 2. For instance, the correlation between the Analytical and Sawthooth columns is 0.989, or the correlation between Secretary Problem and AHP-BWS equals 0.894. So we see that for the suggested techniques of analytical calculations, AHP-BWS, and SP produce very similar solutions, although they require much less effort to elicit data and estimate priorities than the regular BWS estimations using specialized software. Table 2 Choice probability estimates item Analytical Sawtooth Sawtooth Adjusted Secretary Problem(SP) AHP- BWS 1 0.048 0.026 0.051 0.065 0.043 2 0.031 0.013 0.031 0.034 0.058 3 0.256 0.425 0.153 0.166 0.158 4 0.069 0.052 0.108 0.091 0.054 5 0.066 0.067 0.088 0.072 0.087 6 0.016 0.006 0.021 0.019 0.027 7 0.047 0.022 0.083 0.051 0.085 8 0.026 0.002 0.014 0.026 0.031 9 0.061 0.048 0.089 0.081 0.084 10 0.041 0.016 0.047 0.055 0.048 11 0.011 0.001 0.005 0.010 0.012 12 0.068 0.056 0.077 0.085 0.054 13 0.143 0.221 0.135 0.130 0.118 14 0.024 0.013 0.021 0.031 0.030 15 0.028 0.008 0.023 0.026 0.040 16 0.029 0.016 0.022 0.032 0.023 17 0.036 0.008 0.031 0.028 0.049 IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 508 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Table 3 Correlations between the columns of choice probability estimates Analytical Sawtooth Sawtooth Adjusted Secretary Problem AHP- BWS Analytical 1 0.989 0.860 0.934 0.907 Sawtooth 1 0.801 0.888 0.875 Sawtooth Adjusted 1 0.959 0.911 Secretary Problem 1 0.894 AHP-BWS 1 3. Summary This paper considered an extension of the AHP methodology of hierarchical structuring into the area of the Best-Worst Scaling used in marketing research for choice probability estimation of multiple items. The compared items can be divided into several subsets by the criteria of brand, size, packaging, etc., then the BWS balanced design plans and data eliciting are applied to each of the subsets separately, and BWS comparison among the criteria themselves is added. Synthesizing the local subset priorities by the preferences among the criteria yields the global choice probabilities of the items. Another approach is based on the so-called secretary problem known in operations research. The results by the new approaches are close to the standard evaluations, but require much less effort to evaluate. AHP enriches the BWS technique, and the proposed methods are especially useful for working with a large number of items. This approach can be extended to other marketing research techniques. IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 509 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 REFERENCES Belton, V. (1986). A comparison of the analytic hierarchy process and a simple multi- attribute value function. European Journal of Operational Research, 26, 7-21. Doi:10.1016/0377-2217(86)90155-4 Gass, S. (2005). Model world: The great debate – MAUT versus AHP. Interfaces, 35, 308-312. Golden, B.L., Wasil, E.A., & Harker, P.T., editors (1989). The Analytic Hierarchy Process: Applications and studies. Berlin: Springer. Ijzerman, M.J., van Til, J.A., & Bridges, J.F.P. (2012). A comparison of Analytic Hierarchy Process and Conjoint Analysis Methods in assessing treatment alternatives for stroke rehabilitation. The Patient, 5, 45-56. Doi:10.2165/11587140-000000000-00000 Lipovetsky, S. (1996). The Synthetic Hierarchy Method: An optimizing approach to obtaining priorities in the AHP. European Journal of Operational Research, 93, 550-564. Doi:10.1016/0377-2217(95)00085-2 Lipovetsky, S. (2006). Optimal hierarchy structures for multi-attribute-criteria decisions. Journal of Systems Science and Complexity, 22, 228–242. Doi:10.1007/s11424-009-9159- 5 Lipovetsky, S. (2009). Global priority estimation in multiperson decision making. Journal of Optimization Theory and Applications, 140, 77-91. Doi:10.1007/s10957-008-9447-6 Lipovetsky S., (2010). An interpretation of the AHP eigenvector solution for the lay person, International Journal of the Analytic Hierarchy Process, 2, 158-162. Doi: http://dx.doi.org/10.13033/ijahp.v2i2.42 Lipovetsky S. (2011). An interpretation of the AHP global priority as the eigenvector solution of an ANP supermatrix. International Journal of the Analytic Hierarchy Process, 3(1), 70-78. Doi: http://dx.doi.org/10.13033/ijahp.v3i1.90 Lipovetsky S. (2011). Priority eigenvectors in Analytic Hierarchy/Network Processes with outer dependence between alternatives and criteria. International Journal of the Analytic Hierarchy Process, 3(2), 172-179. Doi: http://dx.doi.org/10.13033/ijahp.v3i2.123 Lipovetsky S. (2013). Supermatrix eigenproblem and interpretation of priority vectors in Analytic Network Process. International Journal of the Analytic Hierarchy Process, 5(1),105-113. Doi: http://dx.doi.org/10.13033/ijahp.v5i1.132 Kallas, Z., Gómez-Limón, J.A. & Barreiro, J. (2007). Decomposing the value of agricultural multifunctionality: Combining contingent valuation and the analytical hierarchy process. Journal of Agricultural Economics, 58(2), 218 – 241. Doi: 10.1111/j.1477-9552.2007.00085.x http://dx.doi.org/10.1016/0377-2217(86)90155-4 http://dx.doi.org/10.1016/0377-2217(95)00085-2 http://dx.doi.org/10.13033/ijahp.v2i2.42 http://dx.doi.org/10.13033/ijahp.v3i1.90 http://dx.doi.org/10.13033/ijahp.v3i2.123 http://dx.doi.org/10.13033/ijahp.v5i1.132 IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 510 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Kallas, Z., Lambarraa, F., & Gil, J.M. (2011). A stated preference analysis comparing the Analytical Hierarchy Process versus Choice Experiments. Food Quality and Preferences, 22, 181-192. Doi: http://dx.doi.org/10.1016/j.foodqual.2010.09.010 Meißner, M., Scholz, S.W., & Decker, R. (2008). AHP versus ACA – An empirical comparison. In Preisach, C., Burkhardt, H., Schmidt-Thieme, L., & Decker, R. (Eds.), Data analysis, machine learning, and applications (447-454). Berlin: Springer. Doi: 10.1007/978-3-540-78246-9_53 Meißner, M., Decker, R., & Scholz, S.W. (2010). An adaptive algorithm for pairwise comparison‐based preference measurement. Journal of Multi-Criteria Decision Analysis, 17, 167-177. Doi: 10.1002/mcda.461 Mulye, R. (1998). An empirical comparison of three variants of the AHP and two variants of Conjoint Analysis. Journal of Behavioral Decision Making, 11, 263-280. Doi: 10.1002/(SICI)1099-0771(1998120)11:4<263::AID-BDM301>3.0.CO;2-T Saaty, T.L. (1980). The Analytic Hierarchy Process. New York: McGraw-Hill. Doi: http://dx.doi.org/10.1080/00137918308956077 Saaty, T.L. (1994). Fundamentals of decision making and priority theory with the Analytic Hierarchy Process. Pittsburgh: RWS Publications. Saaty, T.L. (1996). Decision making with dependence and feedback: The Analytic Network Process. Pittsburgh: RWS Publications. Saaty, T.L. (2000). Decision making for leaders. Pittsburgh: RWS Publications. Doi: 10.1109/TSMC.1985.6313384 Saaty, T.L., & Peniwati, K. (2012). Group decision making. Pittsburgh: RWS Publications. Scholl, A., Manthey, L., Helm, R., & Steiner, M. (2005). Solving multiattribute design problems with analytic hierarchy process and conjoint analysis: An empirical comparison. European Journal of Operational Research, 164, 760–777. Doi: http://dx.doi.org/10.1016/j.ejor.2004.01.026 Scholz, S.W., & Decker, R. (2007). Measuring the impact of wood species on consumer preferences for wooden furniture by means of the Analytic Hierarchy Process. Forest Products Journal, 57(3), 23 - 28. Scholz, S.W, Meißner, M., & Decker, R. (2010). Measuring consumer preferences for complex products: a compositional approach based on paired comparisons. Journal of Marketing Research, 47(4), 685–698. Doi: http://dx.doi.org/10.1509/jmkr.47.4.685 Vargas, L.G. (1987). Priority theory and utility theory. Mathematical Modelling, 9, 381- 385. Doi:10.1016/0270-0255(87)90496-9 http://dx.doi.org/10.1016/j.foodqual.2010.09.010 http://dx.doi.org/10.1080/00137918308956077 http://dx.doi.org/10.1109/TSMC.1985.6313384 http://dx.doi.org/10.1016/j.ejor.2004.01.026 http://dx.doi.org/10.1509/jmkr.47.4.685 http://dx.doi.org/10.1016/0270-0255(87)90496-9 IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 511 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Winkler, R.L. (1990). Decision modeling and rational choice: AHP and utility theory. Management Science, 36, 247-248. Doi: http://dx.doi.org/10.1287/mnsc.36.3.247 ADDITIONAL REFERENCES ON CONJOINT, DISCRETE CHOICE, AND BEST-WORST SCALING Ben-Akiva, M., & Lerman, S.R. (1985). Discrete Choice Analysis. Cambridge, MA: MIT Press. Doi: 10.1007/s11336-007-9029-9 Chrzan, K., & Yardley, D. (2009). Tournament-augmented choice-based conjoint. Proceedings of the Sawtooth Software Conference, 163-169, Delray Beach, Florida. Hensher, D.A., Rose, J.M., & Greene, W.H. (2005). Applied choice analysis: A primer. Cambridge, England: Cambridge University Press. Lipovetsky, S., & Conklin, M. (2014a). Best-Worst scaling in analytical closed-form solution. Journal of Choice Modelling, 10, 60–68. Doi: http://dx.doi.org/10.1016/j.jocm.2014.02.001 Lipovetsky, S., & Conklin, M. (2014b). Finding items cannibalization and synergy by BWS data. Journal of Choice Modelling, 12, Doi: 10.1016/j.jocm.2014.08.001. Lipovetsky, S. & Conklin, M. (2015). MaxDiff priority estimations with and without HB-MNL. Advances in Adaptive Data Analysis, 7(1-2), 1-10. Doi: http://dx.doi.org/10.1142/S1793536915500028 Lipovetsky, S., Liakhovitski, D., & Conklin, M. (2015). What's the right sample size for my MaxDiff study? Sawtooth Software Conference Proceedings, 119- 141. Louviere, J.J. (1991) Best-Worst Scaling: A Model for the Largest Difference Judgments, Working Paper, University of Alberta. Louviere, J.J. (1993). The Best-Worst or Maximum Difference Measurement model: Applications to behavioral research in marketing. The American Marketing Association's Behavioral Research Conference. Louviere, J.J., Hensher, D.A., & Swait, J. (2000). Stated choice methods: Analysis and applications. Cambridge, MA: Cambridge University Press. Doi: https://doi.org/10.1017/CBO9780511753831 Marley, A., & Louviere, J.J. (2005). Some probabilistic models of Best, Worst, and Best- Worst choices. Journal of Mathematical Psychology, 49, 464–480. Doi: http://dx.doi.org/10.1016/j.jmp.2005.05.003 http://dx.doi.org/10.1287/mnsc.36.3.247 http://dx.doi.org/10.1016/j.jocm.2014.02.001 http://dx.doi.org/10.1142/S1793536915500028 http://dx.doi.org/10.1016/j.jmp.2005.05.003 IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 512 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 McFadden, D. (1973). Conditional logit analysis of qualitative choice behavior. In Zarembka, P., (Ed.), Frontiers of Econometrics (105-142). New York: Academic Press. McFadden, D. (1981). Econometric models of probabilistic choice. In Manski, C., & McFadden, D., (Eds.), Structural Analysis of Discrete Data (198-272). Cambridge: MIT Press. McFadden, D., & Richter, M.K. (1990). Stochastic Rationality and Revealed Stochastic Preference. In Chipman, J., McFadden, D., & Richter, M.K., (Eds.), Preferences, Uncertainty, and Optimality: Essays in Honor of Leo Hurwicz (151-186). Boulder, CO: Westview Press. Netzer, O., & Srinivasan, V.S. (2011). Adaptive self‐explication of multiattribute preferences. Journal of Marketing Research, 48(1), 140–156. Doi: http://dx.doi.org/10.1509/jmkr.48.1.140 Orme, B.K. (2006). Adaptive maximum difference scaling. Sawtooth Software Research Paper Series. Sequim, WA: Sawtooth Software, Inc. Orme, B.K. (2010). Getting started with Conjoint Analysis, 2 nd edition. Madison, WI: Research Publishers LLC. Sawtooth Software (2007). Technical Paper Series: The MaxDiff/Web v6.0, Sequim, WA: Sawtooth Software, Inc. Sawtooth Software (2014). Technical Paper Series: ACBC Technical Paper. Sequim, WA: Sawtooth Software, Inc. Train, K. (2003). Discrete choice methods with simulation. New York: Cambridge University Press. Wen, C.H., & Koppelman, F (2001). The generalized nested logit model. Transportation Research B, 35, 627–641. Doi: http://dx.doi.org/10.1016/S0191-2615(00)00045-X Wirth R. and Wolfrath A. (2012). Using MaxDiff for evaluating very large sets of items. Proceedings of the Sawtooth Software Conference, 59-78. ON SECRETARY PROBLEM Bearden, J.N. (2006). A new secretary problem with rank-based selection and cardinal payoffs. Journal of Mathematical Psychology, 50, 58-59. Doi: http://dx.doi.org/10.1016/j.jmp.2005.11.003 Chow, Y.S., Moriguti, S., Robbins, H., & Samuels, S.M. (1964). Optimal selection based on relative rank. Israel Journal of Mathematics, 2, 81–90. Doi: 10.1007/BF02759948 http://dx.doi.org/10.1509/jmkr.48.1.140 http://dx.doi.org/10.1016/S0191-2615(00)00045-X http://dx.doi.org/10.1016/j.jmp.2005.11.003 IJAHP Article: Lipovetsky/AHP structuring in best-worst scaling and the secretary problem International Journal of the Analytic Hierarchy Process 513 Vol. 8 Issue 3 2016 ISSN 1936-6744 http://dx.doi.org/10.13033/ijahp.v8i3.332 Ferguson, T.S. (1989). Who solved the Secretary Problem? Statistical Science, 4, 282- 296. Doi:10.1214/ss/1177012498 Freeman, P.R. (1983). The secretary problem and its extensions – a review. International Statistical Review, 51, 189–206. Petruccelli, J.D. (1984). Best-choice problems involving recall with uncertainty of selection when the number of observations is random. Advances in Applied Probability, 16, 111-130. Doi: https://doi.org/10.1017/S0001867800022370 Presman, E.L., & Sonin, I.M. (1972). The best-choice problem for a random number of objects. Theory of Probability and Applications, 17, 657-668. Doi:10.1137/1117078 Rose, J.S. (1982). A problem of optimal choice and assignment. Operations Research, 30, 172–181. Doi: http://dx.doi.org/10.1287/opre.30.1.172 Samuels, S.M. (1985). A best-choice problem with linear travel cost. Journal of the American Statistical Association, 80, 461-464. Stein, W.E., Seale, D.A., & Rapoport, A. (2003). Analysis of heuristic solutions to the best choice problem. European Journal of Operational Research, 151, 140–152. Doi: http://dx.doi.org/10.1016/S0377-2217(02)00601-X Vanderbei, R.J. (1980). The optimal choice of a subset of a population. Mathematics of Operations Research, 5, 481–486. Doi: http://dx.doi.org/10.1287/moor.5.4.481 https://doi.org/10.1017/S0001867800022370 http://dx.doi.org/10.1137/1117078 http://dx.doi.org/10.1287/opre.30.1.172 http://dx.doi.org/10.1016/S0377-2217(02)00601-X http://dx.doi.org/10.1287/moor.5.4.481