A PROCESS FOR DEFINING WHAT THE SUPERVISOR REALLY WANTS Mark Kroll University Of Texas At Tyler Tyler, TX Barbara D. Davis AshayDesai Peter Wright University Of Memphis Memphis, TN Intoduction When a man does not know what harbor he is making for, No wind is the right wind. -Seneca As our citation suggests, people must be provided accurate direction from those they follow, if they are to be productive. In an organizational context, there is a need to communicate to subordinates what the key performance criteria are and how important the various criteria are to the supervisor. Supervisors typically feel they know which criteria are important and what values they might assign to each (Carroll and Schneier 1982). We suggest an objective method for more accurately assessing supervisors' weightings of the perfor- mance criteria used to evaluate their subordinates. We will demonstrate how this approach can define the gaps between supervisors' actual weights, what they thought the weights were they were applying, and the weights the organization feels are appropriate. The Problem Today, virtually every organization of any size attempts to appraise performance. The majority of the appraisal systems in use today focus on broad, trait oriented aspects of employees (llgen, Barnes-Farrell, and McKellin 1993). Typically, these systems have gone through numerous revisions and refinements. Unfortunately, research in this area shows that dissatisfaction with the processes in place is the norm rather than the exception (Landy and Farr 1980). However, it could be that oftentimes the problem is not the system used, but rather the managers' inability of coming to terms with what really matters to them in terms of subordinate performance. Therefore, we will first examine the nature of the problem and what others have said concerning its consequences. We then offer a workable, practical solution: conjoint analysis. Fall 1994 Kroll, et. al.: Defining what the Supervisor Really Wants 99 While the appraiser, as well as the appraisal systems in use have received a great deal of attention, little has been offered to help supervisors get in touch with their own feelings as to which job performance characteristics they most emphasize in subordinate behavior (Ostroff 1993; Pearce and Porter 1986). This is a critically important issue since many of the appraisal systems observed by the authors merely define the dimensions to be evaluated. Frequently, there is little continuity among supervisors or raters as to how to "weight" the various performance dimensions in arriving at a relative ranking of their subordinates (Borman 1987). Raters have definitions of "performance as a whole" which go beyond an employee's performance ofhislher stated job duties. Past research on performance apPfJlisals suggest that the managers go through a complex multiattribute decision process by trading off among a relatively large number of attributes (Budman and Rice 1994; Ilgen, Barnes- Farrell, and McKellin 1993). When performance of an employee is appraised using rating scales, it is done so along certain identified dimensions. The raters judge the value of the identified criteria and usually trade off unfair conditions of some attributes for favOfable conditions of others. Managers' or raters' perceptions play an important role in the complex considerations of alternatives that fall along more than a set of dimensions. Research has provided evidence that varied cognitive processes engaged in, by raters, influence rating accuracy (Ostroff 1993). Refinements in performance appraisals, such as behaviorally anchored rating scales (BARS), have sought to provide more reliable appraisals, less subject to rater bias, though the success ofthese efforts is an open question (Bernardin and Smith 1981; DeCotiis 1977). However, when promotion, demotion, or layoff decisions are made and supervisors are asked to make recommendations, they are likely to base them on individual priorities as to which criteria are more or less important (Bernardin and Smith 1981; DeCotiis 1977). Such decisions are usually made by supervisors after considering the simultaneous and joint effect of ratings along all attribute dimensions. While alternatives have been offered as to how the organization might establish criteria weights, these alternatives offer appraisers little help in coming to understand their own priorities, which may be different from those suggested by the organization. In fact, quite often supervisors are unaware of what their priorities actually are. The organizational method used to establish criteria weights needs to be flexible enough to capture this multiattribute choice problem, which may cause deviations from the criteria weights assigned by the organization. Our focus in this paper is on the raters process of assigning priorities and not on the organizational rating scale used. Some Evidence of Potential Harm Juxtaposed with the supervisors' criteria valuation is the problem experienced by the subordinates of deciding where to concentrate their efforts. Most subordinates neither have the time, talent, or ability to excel in every dimension in which they are evaluated. They must therefore make trade-offs that they anticipate will gamer the greatest value in the eyes of their supervisor. Obviously, if it were possible for subordinates to accurately gauge their supervisors' priorities, their choices would be far more efficient. 100 Journal ofBusiness Strategies Vol. 11, No.2 An important question therefore is whether or not subordinates truly know what their supervisors' priorities are in terms of various performance dimensions. In case they do not, then another question would seem to be whether or not real harm accrues if such misunder- standings do exist. The research to date suggests that the lack of congruence between subordinates and their supervisors, as to the relative importance of various performance dimensions, is a common problem. In addition, this incongruence helps explain a significant portion of the difference between subordinates' assessments of their performance and the ratings they receive from their supervisors (Oberg 1972). More important to the organization, such incongruence has been linked to lower subordinate job satisfaction, performance, and personal and organizational goal attainment. The link between low superior-subordinate goal congruence, job satisfaction and performance has been demonstrated in a number of studies (Ostroff 1993; Pearce and Porter 1986). For instance, ngen, Fisher and Taylor (1979), found a significant gap in job satisfaction levels with those who reported fewer differences between role perceptions and their job descriptions and those who had significant incongruence. These findings were similar to Bernardin (1979), who found a significant, negative relationship between police officer performance and resultant job satisfaction and the level of goal non-congruence. Hobson, Mendel, and Gibson (1981), found the same type of relationship in a later study. From a motivational theory point of view, these findings are quite reasonable (Vroom 1964). For some time, it has been recognized that significant incongruence between superior and subordinate performance expectations will likely diminish the subordinate's belief that effort leads to performance, and consequently, job performance and satisfaction suffers. More accurately, defining what superiors most value in subordinate performance and then communicatingthat information would seem a likely path toward enhancing the subordinate's beliefs that effort invested correctly will lead to performance, goal attainment, and job satisfaction. Related to the problems of lower job satisfaction and performance is the issue of employee development. One of the purposes of appraising performance is to define subordinate training needs. If goal incongruence is a problem, it is possible that what a supervisor may diagnose as a lack of ability may simply be a lack of emphasis based on what the subordinate perceives as a low priority activity (Hobson, Mendel, and Gibson 1981). As mentioned earlier, beyond the problems which might occur for the individual employee, there are also possible negative consequences for the organization. As just mentioned, performance appraisals should tie individual efforts to the goals of the organiza- tion, yet if the priorities of the supervisor are not those of the organization, the appraisal process could be counter-productive (U.S. Government 1978). It might be argued that higher- ups could set weights, thus insming congruence between appraisal outcomes and organiza- tional needs. However, this view fails to consider the inevitable intervention of the appraisers' own values when decisions are made regarding employees' futures (e.g., promotion, demotion or transfer decisions). Fall 1994 Kroll, et. al.: Defining what the Supervisor Really Wants 101 The Research Gap It is of critical importance to the organization that supervisors exhibit high inter-rater consistency in assigning weights to the same job. Additionally, it is imperative that the weights supervisors assign to job performance criteria correlate with the goals of the organization. The success an organk:ation has in utilizing and developing its talent depends, at least in part, on the congruence between organizational and individual priorities. Insuring such congruence should be one of the goals of performance appraisals. While a great deal of attention has been given to aiding organizations in achieving inter-rater reliability for a given performance profile, organizations have been provided little help in assessing and evaluating raters' weightings of the performance criteria included in the evaluation processes used. Research has provided evidence that varied cognitive processes engaged in by raters influence rating accuracy. As mentioned earlier, various methods are available to aid organizations establish criteria weights. Methods such as allowing a panel of judges to set the weights, measuring the correlation of criteria with "performance," or simply weighting each criteria equally. The problem with each of these is that they do not explore the appraiser's own values and priorities. Appraisers quite simply may be unaware of how heavily they weight one criteria in relation to others. It is important to recognize that all the criteria included in an appraisal will carry a positive value (Le., few supervisors will likely not want subordinates to do well on a given criterion). Yet, inevitably, supervisors will see some as more important than others. Consequently, there is a need to apply a technique to uncover the latent bias, sometimes unknown to the raters themselves. Given the potential benefits, a practical, objective means for assessing supervisors' performance priorities would seem very desirable. Such measurements would allow supervisors to come to know themselves and communicate their priorities to their subordi- nates. Additionally, the organization could determine whether their supervisors' priorities are in line with its own. While attempts have been made to define weights using techniques such as multiple regression analysis (Hobson, Mendel, and Gibson 1981), conjoint analysis offers an alternative prospective to defining criteria weights incorporating supervisors' performance priorities. A Possible Solution: Conjoint Analysis We wish to accomplish four tasks in this section. First, conjoint analysis is briefly described and illustrated. Second, adaptive conjoint analysis is introduced. Third, an illustration of the use of adaptive conjoint analysis in defining supervisor performance criteria weights in an actual organization is presented. Finally, the results are evaluated in terms of how they vary across departments and organizational levels, and what the consequences of such variances might be. Conjoint Analysis Though not previously applied to the area of performance appraisal, conjoint analysis has been used in marketing research for quite some time (Green and Rao 1971). However, in the last decade, increased availability of software packages has made it easier for the application of conjoint methodology to areas such as corporate strategy and human resource 102 Journal ofBusiness Strategies Vol.U, No.2 issues pertaining to compensation analysis, litigation and determining employee benefit preferences (Green, Krieger, and Agarwal 1991; Riquelme and Rickards 1992). "Conjoint" refers to a process of measuring the relative values people assign to attributes or features (typically product features when used in market research) when they consider them together rather than individually. This allows the researcher to determine the relative importance of various product features as they are considered within the context of a set of features (or as a whole product package) by the person being asked to make the evaluation. This technique has traditionally been used in product development or positioning. In an attempt to determine the optimal combination of product features, potential buyers are presented with different versions of the product, with varying feature combinations. By analyzing consumer rankings of the various product offerings from most to least desirable, conjoint analysis defines how valued the various features are by the potential buyer or how much ......eight each feature carries in the decision process (Green and Yoran 1975). It is perhaps obvious that the same approach in defining product feature weights could also be used to define weights for subordinate performance criteria. What follows is a brief (and somewhat simplistic) illustration of the use of conjoint analysis in performance appraisal, a more extensive illustration will be presented later. Assume that a firm wishes to understand how heavily its supervisors weight each of three performance criteria: delegating skills, communication skills, and decision-making skills. Rather than simply asking each supervisor to rate the criteria individually, (e.g., "which of the three criteria is most important to you?") they are asked to rank order from least to most preferred, all of the possible combinations ofthe three criteria, with four performance levels each (e.g., excellent, good, fair, poor), for a total of 64 possible combinations. Table 1 lists the possible choices, and the hypothetical selections made by a supervisor. As there are three criteria involved, each ranking presented represents a combination of all three. For instance, the value -13 circled in Table 1 indicates that the supervisor has ranked as 13th from the bottom the combination of: poor communication, fair decision making and excellent delegating skills. Conjoint analysis attempts to identify consistencies in the supervisor's judgment of the relative desirability of the various criteria and the importance placed on the various levels of performance (e.g., "excellentvs. "good). In effect, conjoint analysis determines the influence of each criterion on the supervisor's ranking of the performance profiles. The technique permits determination ofwhich criterion the supervisor feels is of greatest importance, which is second, etc. Additionally, the importance of one level of performance versus another is ascertained (e.g., how much more valued is "excellent" versus "good"). Each criterion level is assigned a numerical weight, or measure of relative significance, so that when the weights of each criterion for each performance profile are summed, the result represents a score for that performance profile which should closely parallel the supervisor's original ordering of the performance profiles. Fal11994 Kroll, et. al.: Defining what the Supervisor Really Wants Table 1 Supervisor's Rankings of Performance Profiles Communication SkiDs 103 E G F P D E eM I A S K I I ON NG E G F P E G F P E G F P E G F P E 64* 58 40 16 61 56 37 14 59 49 34 11 47 44 31 8 G 63 57 39 15 60 53 36 12 52 48 33 9 46 42 30 6 F 62 55 38@ 54 51 35 10 50 45 32 7 43 41 29 2 P 28 26 20 5 27 25 19 4 24 22 18 3 23 '2.1 17 1*· E G F P *•• Delegating Skills indicates the most preferred alternative indicates the least preferred alternative Performance Levels: E- Excellent F-Fair G-Good P - Poor This table presents an illustrative ranking of aU 64 possible performance profiles made by a supervisor. The 64th profile represents the one which the supervisor most favored while the lst represents the one the supervisor least favored. Table 2 Part Weights Performance Factors Delegating Skills Communication Skills Decision Making Skills Excellent 47.0 Excellent 89.0 Excellent 52.0 Good 19.0 Good 47.0 Good 21.0 Fair 7.0 Fair 16.0 Fair 2.0 Poor 0.0 Poor 0.0 Poor 0.0 This table presents the "part weights" or measures of relative importance the supervisor holds for the various criteria levels, based on the choices presented in Table 1. 104 Journal ofBusiness Strategies Vol. II, No.2 Using the supervisor rankings provided in Table 1, Table 2 provides the conjoint analysis generated weights for each level of the three performance criteria. It is quite easy to identify which of the three criteria is of greatest consequence in influencing this supervisor's ratings of subordinates, in this case--communication skills. In fact, this particular supervisor values "good" communication skills (worth 47 points), as highly as "excellent" communicatioon skills, while delegating skills (worth 47 points) is the criterion the supervisor least values. A subordinate who did well in terms of hislher communication skills rating but poorly on delegating skills would be considered a much better performer by this supervisor than had the subordinate done best on delegating skills and worst on communication skills. In addition to allowing us to assess the relative importance to a supervisor of each criterion, as is apparent from Table 2, conjoint analysis permits assessment ofhow important one level ofperformance is versus another (e.g. "excellent" performance for a criterion versus "good" performance). Simply put, given that numerical weights are defined at every level of each criterion, it is possible to measure the gap between levels of the criterion. We can tell how important a supervisor feels one level of performance is in relation to another. Additionally, since the various weights of each criterion can be added for any possible performance combination, the overall difference between one performance profile and another can be fairly and precisely quantified. For instance, using the data in Table 2, assume we wished to know how this particular supervisor would rate a subordinate with poor delegating skills, fair communication skills, and good decision-making skills versus a subordinate with good delegating skills, fair communication skills and poor decision-making skills. By adding the appropriate weights for each of the criterion, we arrive at total profile scores. The fIrst subordinate would be ranked slightly higher (0 + 16 + 21 = 37) than the second subordinate (19 + 16 + 0 = 35). The basic idea behind the conjoint analysis is to decompose a set of overall responses to multiattribute alternatives, which can then be stratified. This stratification makes it easier to understand the intricate decision processes a supervisor goes through during performance ratings. Afirm using either rating scales, BARS, or behavioral expectations scales could very easily apply conjoint analysis, using the already established criteria weights. Applying conjoint analysis in organizations using management by objectives or some other approach would obviously not be so simple, as shared criteria would first have to be identifled. Both the earlier illustration, as well as the one that follows are based upon a rating scale approach. This is simply because rating scales are a very widely used evaluation technique. Innovations in Conjoint Analysis The obvious problem with using traditional conjoint analysis (as just illustrated) to measure supervisor performance preferences is the number of profiles which would have to be ranked based on the typical evaluation instrument. For instance, a rating instrument consisting of five criteria with four possible levels of performance would result in 625 (5 x 5 x 5 x 5) combinations. No supervisor could be expected to rank so many performance profiles. Fortunately, recent interactive software developments now make it possible to accurately assess a supervisor's preferences by asking him/her to make less than 25 evaluations, taking about 10 minutes. Such software was used in the following illustration Fall 1994 Kroll, et. al.: Defining what the Supervisor Really Wants 105 and is now readily available (here, the "Adaptive Conjoint Analysis Program" developed by Sawtooth Software was used). While a detailed discussion of the algorithms employed is beyond the scope of this paper, very simply put, weights are assigned as a result of a two-step process. First, supervisors are simply asked to indicate how important each criterion is on a Likert-type scale (e.g., 1 to 10, with 10 being assigned to very important criteria). Incorporating this information, the software presents the supervisor with a series of performance profile contrasts (see Exhibit 1). Exhibit 1 Example of Questions Asked in the Adaptive Conjoint Analysis Interview Instructions: Please indicate which of the two performance profiles presented below you prefer most by typing one of the values on the left hand side of the screen. Strong Prefer Top 1 2 3 4 5 6 7 8 9 Strong Prefer Bottom Unacceptable Planning/Organizing Skins (unable to plan and implement) Below Average Interpersonal Skills (occasionally underperforms) Below Average Technical Skills (needs improvement in technical skills) Above Average Technical Skills (exceeds requirements) Below Average Supervisory SkiDs (needs improvement) OR Below Average Planning/Organizing Skills (needs improvement) Above Average Interpersonal Skills (frequently displays above expected interpersonal skills level) Unacceptable Technical Skills (doesn't meet minimum requirements) Below Average Personal Skills (needs improvement in dependability, creativity and adaptability) Unacceptable Supervisory Skills (cannot supervise the work of others) 106 Journal ofBusiness Strategies Vol. 11, No.2 By favoring one profile over another, or expressing indifference to the two profiles, the supervisor provides indications of which performance criteria are favored, and how much those criteria influence his/her preferences. The software continues to present profile contrasts until a clear picture of criteria weights is developed. Exploring Criteria Weights in the Organization In order to illustrate the potential benefits of using interactive conjoint analysis in conjunction with the performance appraisal process, the technique is applied in a fairly large, regional bank ($500 million in total assets). The purpose is threefold: to illustrate what the results of such analysis are like for an individual supervisor, to illustrate how the results can be compared and contrasted among supervisory ranks, and to assess how upper-level management views the criteria weights their subordinate managers are applying vis-a-vis organizational priorities and goals. . The organization studied was, as mentioned, a relatively large bank which is a component of a very large holding company. The bank has developed three performance appraisal instruments. One is used by first-line supervisors to evaluate operative level employees (mostly non-exempt). A second is used by mid-level managers to evaluate first- line supervisors (these are all exempt employees). The third is used by the CEO to evaluate middle management (who typically carry the title vice president or senior vice president). For the sake of brevity, this study will focus only on criteria weights middle managers apply in evaluating first line supervisors and criteria weights the CEO applies in evaluating middle management. Middle Managers Middle managers of the bank are asked to evaluate their first line supervisors using an evaluation instrument consisting of five attributes or criteria. What follows is a listing of the criteria, as well as their descriptions taken from the performance appraisal documents used by the bank. 1. Planning and organizing - this criterion refers to the subordinate supervisor's ability to define objectives, monitor progress toward objectives, prioritize tasks, set schedules and adequately document departmental progress (ie. securing authorization, making appropriate filings). 2. Technical skills - this criterion refers to the supervisor's ability to keep up-to-date technically with regard to banking practices, analyzing information and evaluating findings critically, and using the information and expertise he/she should possess to make sound decisions. 3. Interpersonal skills - this criterion refers to the supervisor's ability to communicate effectively, coordinate the work of the department, and represent his/her employees to others within the. bank and the bank to the community. 4. Personal Skills - this criterion refers to the supervisor's dependability, innovativeness, and adaptability to change (willingness to try new approaches to problems/situations). 5. Supervisory skill- this criterion refers to the development of human resources in his/ her department through coaching and delegating, as well as the supervisor's ability to Fa111994 Kroll, et. al.: Defining what the Supervisor Really Wants 107 motivate the employees in hislher unit toward the accomplishment of departmental objec- tives. Middle managers can assign one of four possible scores to their supervisors for each of the criteria just listed: superior, above average, below average, and unsatisfactory. They are required to also write out brief statements justifying their scoring for each ofthe four criteria. It should be pointed out that it is not the purpose of the paper here to render judgment on the relative validity or appropriateness of the criteria used by the institution. The purpose is simply to illustrate how conjoint analysis can be adapted to an appraisal scheme. Table 3 presents criteria weight profiles for four of the bank's mid-level managers. The weights for all five factors total five hundred points (averaging 100 points for each criteria). Table 3 Criteria Weights for Middle Managers Manager A ManagerB ManagerC ManagerD Planning/Organizing Superior 60.0 61.0 70.0 51.0 Above Average 18.0 50.0 47.0 26.0 Below Average 9.0 27.0 11.0 16.0 Unacceptable 0.0 0.0 0.0 0.0 Interpersonal Superior 40.0 37.0 33.0 55.0 Above Average 37.0 20.0 32.0 41.0 Below Average 31.0 7.0 16.0 23.0 Unacceptable 0.0 0.0 0.0 0.0 Technical Superior 93.0 45.0 27.0 89.0 Above Average 60.0 41.0 27.0 59.0 Below Average 12.0 18.0 16.0 17.0 Unacceptable 0.0 0.0 0.0 0.0 Personal Superior 53.0 30.0 24.0 61.0 Above Average 34.0 30.0 34.0 32.0 Below Average 14.0 9.0 21.0 16.0 Unacceptable 0.0 0.0 0.0 0.0 Supervisory Superior 34.0 50.0 66.0 8.0 Above Average 5.0 58.0 53.0 6.0 Below Average 0.0 27.0 23.0 0.0 Unacceptable 0.0 0.0 0.0 0.0 TOTAL 500.0 500.0 500.0 500.0 108 Journal ofBusiness Strategies Vol. 11, No.2 As is apparent from a review of Table 3, there are some significant differences in the weights assigned by the supervisors. Most notable are the differences in importance placed on technical skills (ranging from 93 for manager A, to 27 for manager C) and supervisory skills (manager C places fairly great significance on this criteria, while manager D sees it as largely irrelevant). It is also worth noting that, in many instances, the managers consider "above average" performance on certain criteria as being just as good as "superior" performance. For instance, manager C weights above average interpersonal skills almost as heavily as "superior" interpersonal skills. The same is true of manager B's weighting of personal skills performance. In terms of the manager's reaction to the findings, there was a combination of confirmation of expectations and genuine surprise. Manager D was most surprised at having given such little weight to supervisory skills, while manger A expressed concern at how heavily he was weighting technical skills at the expem:e of personal and supervisory skills. The Chief Executive Officer The CEO of the bank does all the evaluations of the middle level managers. He uses an instrument that differs somewhat from that used by his subordinates to evaluate supervisors; it consists of only four criteria: planning and organizing, interpersonal, personal, and supervisory skills. The results of the conjoint analysis of these four criteria are presented in Table 4 below. Table 4 Criteria Weights for the CEO Planning/Organizing 64.0 44.0 23.0 0.0 Superior Above Average Below Average Unacceptable Interpersonal Superior 39.0 Above Average 33.0 Below Average 7.0 Unacceptable 0.0 Personal Superior 41.0 Above Average 33.0 Below Average 12.0 Unacceptable 0.0 Supervisory Superior 46.0 Above Average 36.0 Below Average 23.0 Unacceptable 0.0 TOTAL 400.0 Fall 1994 Krol~ et. al.: Defining what the Supervisor Really Wants 109 One of the more interesting observations concerning the CEO's weightings is the relative balance. The CEO displays much greater uniformity ofweights when contrasted with his subordinates' weights. When asked what possible cause might explain this, he responded that he expects his subordinate managers to be fairly balanced, generalists managers. He also mentioned that, at the middle management level, planning and organizing are quite important to him. In fact, a review of Table 4 reveals that planning was the most heavily weighted criteria. Overall, the CEO was only surprised by the relatively heavy weight assigned to superior supervisory skills in relation to other criteria. Discussion After completing conjoint analyses for all of the managers in the bank and sharing the results with them, a discussion was held with the CEO and the middle level managers. The purpose was to allow the CEO and his immediate subordinates to explore both the weights he received, as well as those received by each of his top executives on the specified performance dimensions. What follows is an excerpted summary of the observations resulting from the discussions. Manager A is the chief lending officer of the bank. As a review of Table 3 reveals, he tended to be the most concerned with technical skills as well as planning and organization. The discussion resulted in the conclusion that this might be due to the difficult banking climate that existed in this particular market, with many defaults and problem loans. This officer's emphasis came as a surprise to others who thought the ability to represent the bank to the business community, which would be measured as "interpersonal skills" on the bank's instrument, would be most critical. However, manager A later explained how much the last few years of banking difficulty had changed his view of the business, and obviously his view of subordinate behavior. Manager B, who heads operations, had perhaps the most balanced criteria weight profIle of the four managers. He also is responsible for more supervisors and more diverse activities--ranging from maintenance to safety deposit box operations. As a consequence, he felt that his weightings reflect the balance required to lead such a diverse unit. However, it was also recognized that manager B was probably not as concerned as he should have been about technical skills, as his unit consisted ofsuch technical activities as data processing and automatic teller machines. Manager C, who heads the trust operations was at a loss to explain why he placed such little emphasis on technical skills and yet weighted planning and organizing so heavily. This was of special concern given the fiduciary responsibilities of the unit. After some deliberation, it was suggested that the strong emphasis on planning and organizing reflected manager C's efforts to get his department heads to "act more like managers and less like clerks" (manager C had held his position for only a year at the time ofthis study). Manager D heads the credit administration department. This department is responsible for such functions as credit analysis, note and loan document processing, collateral inspection and valuation, and handling loans in foreclosure. According to manager D, the technical skills of the people who handle these various functions determines success or failure. As a result, he was not disappointed at the very minor concern he has for supervisory skills or personal skills. 110 Journal ofBusiness Strategies Vol. 11, No.2 There was a comment concerning his failure to develop his subordinates' supervisory skills, with an eye toward their promotion. Overall the results contained both some surprises and some confrrmations of what was anticipated before the analysis. Mterreviewing the results with his vice-presidents, the CEO did express a desire for managers A and B to place greater emphasis on developing their subordinate managers' supervisory skills. There was also some discussion of the fact that managers Band C oftentimes weighted "above average" performance as heavily as "superior." For instance, manager B weighted the two levels equally on the personal criteria, while manager C weighted "above average" more heavily than "superior" on the personal criteria. After reviewing the appraisal instrument, they concluded that a subordinate who was above average in dependability and adaptability was fulf'Illing the position adequately, that the marginal difference between "superior" and "above average" levels on this criteria was not of real importance to them. Conclusion Our purpose here has been to present an innovative method for measuring supervisors' weights of the different criteria used in evaluating their subordinates. Traditionally, rating scales or other techniques have been used for appraisal without much thought given to how evaluators view the criteria which make up the instrument. Heretofore, those organizations which did seek to determine how their supervisors value the criteria being used had to either resort to regression analysis techniques or simply ask the supervisors to rank their priorities from fIrst to last. While the rust approach is likely to be too complex for most organizations to attempt, the latter really does not get at the issue of how much one performance dimension a supervisor will trade for another. It is quite possible that some organizations may have more elaborate performance appraisal systems than the above example. However, there are meaningful differences within managerial perceptions, which can affect the ratee's perfor- mance appraisal. Periodic checks need to be introduced in the organization to detect any deviations which have crept into the performance appraisal system. Conjoint analysis offers both an efficient and effective means for exploring supervisors' values and ascertaining what it is they truly prize in their subordinates' performance. With the introduction ofcommercial conjoint computer packages such as Bretton-Clark's Conjoint design, Con Analyzer, SIMGRAF, and Adaptive Conjoint Analysis (Green, Krieger, and Agarwal 1991), it is easier to implement such periodic checks within the performance appraisal system without much difficulty. In fact, software innovations ofthe past few years permits this analysis to be done in a matter of minutes and provides easily interpreted results. This information can be benefIcial for the supervisor, hislhersubordinates, and the organization. The supervisor may, for the fIrst time, come to understand not only which criteria they value the most, but how much so. Subordinates benefIt (if the results are shared with them) by having a clearer picture of just what the supervisor really wants. The organization, also for the rust time, can determine whether or not what its supervisors are looking for in subordinate behavior matches its own needs. After years of attempting to fIx the performance appraisal process, maybe it is time to understand and agree upon just what it is we are looking for and value in subordinate behavior. Conjoint analysis may offer a workable means for doing just that. Fa111994 Kroll, et. al.: Defining what the Supervisor Really Wants References 111 Bernardin, H. J. 1979. "The Predictability of Discrepancy Measures of Role Constructs," Personnel Psychology 32 (Spring): 139-153. Bernardin, H. and P. Smith. 1981. "A Clarification of Some Issues Regarding The Development And Use of Behaviorally Anchored Rating Scales," Journal of Applied Psychology 66 (August): 45-46. Borman, W. 1987. "Personnel Constructs, Personnel Schemata, and 'Folk Theories' of Subordinate Effectiveness: Explorations in the Army Officer Sample," Organizational Behavior and Human Decision Processes 40 (December): 307-322. Budman, M. and B. Rice. 1994. "The Rating Game," Across the Board 31 (February): 34- 38. Carroll, S. J. and C. Schneier. 1982. Performance Appraisal and Review Systems. Glenview, IL: Scott, Foresman. DeCotiis, T. 1977. "An Analysis of External Validity and Applied Relevance of Three Rating Formats," Organizational Behavior and Human Performance 19 (August): 247- 266. Green, P., A. Krieger, and M. Agarwal. 1991. "Adaptive Conjoint Analysis: Some Cavaets and Suggestions," Journal of Marketing Research 28 (May): 215-225. Green, P., and V. Rao. 1971. "Conjoint Measurement for Quantifying Judgmental Data," Journal of Marketing Research 8 (August): 355-363. Green, P. and W. Yoran. 1975. "New Way to Measure Consumer Judgements," Harvard Business Review 53 (July-August): 107-119. Hobson, C., R. Mendel, and F. Gibson. 1981. "Oarifying Performance Appraisal Criteria," Organizational Behavior and Human Performance 28 (October): 164-188. llgen, D. R., J. Bames-Farrell., and D. McKellin. 1993. "Performance Appraisal Process Research In The 1980s: What Has It Contributed To Appraisals In Use?" Organiza- tional Behavior and Human Decision Processes 54 (April): 321-368. llgen, D. R., C. D. Fisher, and M. S. Taylor. 1979. "Consequences ofIndividual Feedback on Behavior In Organizations," Journal of Applied Psychology 64 (August): 349-371. 112 Journal ofBusiness Strategies Vol. 11, No.2 Landy, F. and J. Farr. 1980. "Performance Ratings," Psychological Bulletin 87 (January): 72-107. Oberg, W. 1972. "Make Performance Appraisal Relevant," Harvard Business Review 50 (January-February): 61-67. Ostroff, C. 1993. "Rater Perceptions, Satisfaction and Performance Ratings," Journal of Occupational & Organizational Psychology 66 (December): 345-356. Pearce, J. and L. Porter. 1986. "Employee Responses to Formal Performance Appraisal Feedback," Journal of Applied Psychology: 71 (May): 182-211. Riquelme, H. and T. Rickards. 1992. "Hybrid Conjoint Analysis: An Estimation Probe in New Venture Decisions," Journal of Business Venturing 7 (November): 505- 518. Seneca, Epistulae ad Lucilium. Uniform Guidelines on Employee Selection Procedures. 1978. Washington, DC: U.S. Government Printing Office. Vroom, V. 1964. Work and Motivation. New York: Wiley. A Process for Defining What the Supervisor Really Wants