Microsoft Word - 1982 8-2 Validity of the Assessment Centre.doc THE VALIDITY OF THE ASSESSMENT CENTRE METHOD AS A PREDICTOR OF MANAGERIAL POTENTIAL 1 I.G. SAKINOFSKY I. V AN W. RAUBENHEIMER NATIONAL INSTITUTE OF PERSONNEL RESEARCH DEPARTMENT OF INDUSTRIAL PSYCHOLOGY RAND AFRIKAANS UNIVERSITY OPSOMMING In hierdie studie is 'n ex post facto ondersoek gedoen na die geldigheid van 'n besondere takseersentrum soos dit deur 'n versekeringsmaatskappy geadministreer word. Daar is vasgestel dat die besondere takseersentrum 'n geldige identifiseerder van bestuurspotensiaal is. Daar is voorgestel dat die organisasie groter voordeel uit die takseersentrum kan put indien dit nie in isolasie beskou word nie, maar tesame met addisionele insette in terme van die huidige funksionering van die organisasie in werking gestel word. Die takseersentrum behoort ook 'n integrale deel van die inligtingsvloei deur die organisasie te vorm. The Assessment Centre is a process involving a number of participants who take part in a series of individual and group exercises administered by a specialist, while being observed by several specially trained impartial observers. The prime objective of the Assessment Centre is the assessment of managerial personnel, and consequently the exercises in which the participants are involved simulate the problems and challenges of jobs for which their potential is being considered. Such jobs are referred to as target level jobs. By simulating a futuristic job situation, the Assessment Centre provides empirical information which can be used to predict an individual's performance in a future situation. An advantage is that information thus derived can be used to prepare individuals for the future by identifying weak points which can be developed. The objective of this study was to determine the validity of an Assessment Centre as a predictor of managerial performance in an insurance association. Numerous studies have been conducted demonstrating the validity of the Assessment Centre in a variety of organizational settings (Finkle, 1976; Byham, 1971). However, states 1 Requests for reprints should be sent to the first author. Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 2 Moses (1975), each user must ascertain the validity of the program as applied in each particular organization. An additional comment by Finkle (1976) lends weight to Moses' point of view. Finkle claims that "the variations in objectives, climate, situational factors, staffing, etc. that characterize Assessment Centre programs should alert professionals themselves to the dangers of universal or overly parsimonious approaches to development, analysis and verification" (p. 880). He refers to the danger of too much being done in conventional ways and on conventional lines. Hence the need for establishing the validity of this particular Assessment Centre - not only in terms of verifying the model used in the current context, but also in terms of justifying a substantial financial outlay on the part of the organization administering it. METHOD The investigation of the effectiveness of the Assessment Centre as a method of predicting managerial performance was conducted in two phases. In the first phase personality orientated criteria were used. In the second phase the evaluation was based on job performance criteria. Phase 1 An independent measure of personality was required to rate participants on dimensions common to their management positions, but independent of Assessment Centre variables so as to avoid a circular finding. For this purpose the following instruments were used: − The Leadership Behaviour Description Questionnaire (LBDQ) Form xii (Stogdill, 1963) in order to determine the leadership behaviour of the individual. This questionnaire was completed in each case by the individual's immediate subordinate. − The Mach V Attitude Inventory (Christie & Creis, 1970) indicates a person's general strategy for dealing with people. − The Purpose-in-Life Test (Crumbaugh, 1968) which is designed to measure the degree to which a person experiences a sense of meaning and "purpose in life", and which relates to his degree of fulfilment. These measures were combined into a "Manager Personality Profile" and profiles were obtained from 82 managers employed by the subject company, on a middle management Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 3 level, who had attended the Assessment Centre. Profiles were also obtained from a control group of 87 managers drawn from external companies in the insurance and finance industry who occupied managerial positions on a level comparable to those employed by the subject company. It was hypothesized that the predictive accuracy of the Assessment Centre would be reflected in a statistically significant relationship between the Assessment Centre ratings of the participants and the Manager Personality Profile of the experimental and control groups. Phase 2 Having established that the Assessment Centre presented a classification which effectively discriminated amongst its participants, it was then desired to relate this differential classification to performance on the job. The ex post facto nature of the investigation implied that the performance of those managers already occupying target level positions was analysed. As mentioned earlier, target positions are those jobs at the level in the organization that the Assessment Centre is designed to simulate, i.e. those positions for which participants at the Centre are being assessed in terms of promotion at the time of the Centre. These managers, who numbered fifteen, were classified into three groups using performance at the Assessment Centre as an independent variable. Actual performance on the target level job of those participants whom the Assess- ment Centre reviewed as favourable (F) was compared with the actual performance on the target-level job of those whom the Assessment Centre reviewed unfavourably (U). These findings were related to a control group (N) consisting of job incumbents of target level jobs who had never attended an Assessment Centre. The instrument used to analyse the performance of the managers is known as a "Record of Management Practice" and represents a breakdown of how each manager spends his working day. The method of data collection was the diary method. For a period of five working days the fifteen subjects kept a written record of their movements and conversations. The analysis of this data was based on the work of Mintzberg's theory of the "Manager's Working Roles" (Mintzberg, 1973). It was hypothesized that there would be a statistically significant relationship between a subject's particular "roleset" (Mintzberg, 1973), and the fact that he received either a favoura- Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 4 ble rating on the Assessment Centre (F), an unfavourable rating on the Assessment Centre (U), or did not attend an Assessment Centre (N). RESULTS AND DISCUSSION Phase 1 In order to identify any relationship between the Manager Personality Profile and the Assessment Centre data, the following steps were undertaken: Correlations were drawn between the Assessment Centre ratings of the Experimental Group (those subjects who had attended an Assessment Centre) and the Manager Personality Profiles (per dimension) of the Experimental Group. These Manager Personality Profiles were also correlated with those of the Control Group. Table 1 represents the Manager Personality Profile means for the Experimental and Control Groups. TABLE 1 MANAGEMENT PERSONALITY PROFILE MEANS FOR EXPERIMENTAL (IN TERMS OF ASSESSMENT CENTRE INTERVALS) AND CONTROL GROUPS Assessment Centre Groups F FA A A/UF UP Control Group LBDQ DIMENSIONS: Representation 5,0 5,4 5,3 5,2 5,0 5,28 Reconciliation 5,5 5,3 5,3 5,3 4,7 5,19 Tol. Uncertainty 9,1 8,9 8,4 9,3 9,4 9,04 Persuasion 10,6 10,0 10,3 9,5 9,8 9,84 Structure 10,0 10,3 10,9 10,2 10,4 10,64 Tol. Freedom 11,0 10,3 10,1 10,5 10,5 9,51 Role Assumption 10,6 10,3 10,9 10,2 10,0 10,68 Consideration 9,9 10,3 10,2 9,8 10,5 9,96 Prod. Emphasis 8,5 8,9 9,7 9,3 9,4 9,50 Predictive Acc. 5,0 4,9 5,0 5,0 5,1 4,73 Integration 4,9 4,9 5,2 4,9 5,0 5,23 Superior Orient. 10,1 10,4 10,8 10,8 10,3 10,40 MACH V: 100 95 93 95 93 96,6 Purpose-in-life 116 115 110 113 109 114,4 Legend: F = Favourable A = Average UF = Unfavourable The intercorrelations between dimensions of the Manager Personality Profiles within Assessment Centre intervals were also examined. The Pearson correlation coefficients were Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 5 calculated with the level of significance being ,01. The chi square test of independence was applied to provide a view of the interrelationship of the LBDQ (Leadership Behaviour Description Questionnaire) dimensions, taken as a whole, to the various Assessment Centre Internal Groups as well as the Control Group. This was to examine any aggregate differences which may have emerged between the various groupings of the subjects. The initial correlations showed no significant trend across the different groups, i.e. there was no evidence of any relationship between membership of any of the Experimental Groupings or Control Group and the scores per dimension of the Manager Personality Profiles. A finding confirmed by a non-significant Pearson correlation coefficient in each case. Similarly, the chi square test of independence produced calculated chi square scores which in no way exceeded the tabled chi square scores, thereby confirming the finding that there is little variation between the Assessment Centre Groupings of the Experimental Group and the Control Group, in terms of their mean LBDQ norms. It was found that the intercorrelation between the dimensions of Consideration and Production Emphasis varied significantly between the unfavourable performers on the Assessment Centre (UF) and the favourable performers (F). This variation is represented in Table 2. TABLE 2 THE INTERCORRELATION OF THE DIMENSIONS, CONSIDERATION AND PRODUCTION EMPHASIS FOR THE EXPERIMENTAL GROUPINGS AND THE CONTROL GROUP Unfavourable Performer (UF) on Assessment Centre Favourable Performer (F) on Assessment Centre Control Group ,80 ,15 ,36 As intercorrelation between Consideration and Production Emphasis of ,80 was found amongst the nine subjects in the (UF) group, with the level of significance being ,05. On the other hand, the intercorrelation between the same two dimensions for the (F) group of subjects was ,152, and amongst the Control Group an intercorrelation of ,363 was obtained between the two dimensions. Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 6 The strong negative intercorrelation amongst the (UF) group implies that the managers in that group stress one dimension (either Consideration of Production Emphasis) at the expense of the other, whereas this relationship does not occur amongst members of the (F) group. It was also found that the intercorrelation between the Mach V and the Purpose-in-life scales varied significantly between the (F) and the (UF) groups. Among the subjects in the (UF) group an intercorrelation of -,41, significant at the ,01 level, was found between the two variables. An intercorrelation of -,197 was observed amongst the (F) group, whilst an intercorrelation of -,266 significant at the ,01 level was found between Mach V and Purpose- in-life scales for the Control Group. Both these pairs of dimensions revealed intercorrelations amongst the (UF) group, i.e. those who had received unfavourable ratings at the Assessment Centre, which were negative and which were more pronounced in this group than in any other. The negative correlation between Consideration and Production Emphasis is not a novel finding. However, the finding that a negative correlation existed between the Machiavellian score and the Purpose-in-life score of those managers who received a relatively low Assessment Centre rating, is of considerable interest. It is hypothesized that an inadequate manager, as defined by his Assessment score, is liable to compensate for his inadequacy in either of the two directions, whereas a competent manager is not faced by such a crisis. An inadequate manager may either adopt a high Machiavellian stance on the one hand, or he may turn towards a quest for meaningfulness of action and self-justification at the cost of, or with less emphasis on, goal achievement. The Purpose-in-life concept has been found to be related to the construct of depression, and this suggests that a depressed individual might have a low Purpose-in-life score (Crumbaugh & Maholick, 1973). Possibly it was this same factor that might have incurred a low performance level on his part during an Assessment Centre, and possibly he compensates by developing a Machiavellian outlook as a form of defence to help him adapt to changes around him. Furthermore, individuals with high Mach V scores have been found to be extremely resistant to social influence and to possess a cool detachment making them less emotionally involved with others (Christie & Creis, 1970). This does not suggest that they are more hostile or vicious, but rather that this cool detachment is a barrier which they erect because of Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 7 feelings of insecurity and anxiety, and to aid them in achievements of their goal despite these feelings of self doubt. This finding might also point to the fact that there might just be an optimum balance between the degree of sensitivity and the degree of manipulatibility a manager should exhibit and this Assessment Centre has managed to detect a swing to either extreme. The theoretical implications aside, the fact that certain intercorrelations were detected within one group that were not present in another, led to the conclusion that the groups were distinct from one another. Consequently the null hypothesis was rejected, and it was established that there is a relationship between the Manager Personality Profile and Assessment Centre scores; such that the Assessment Centre does distinguish between managers. Phase 2 Having established that the Assessment Centre does effectively distinguish between different managers, the subsequent objective was to relate this distinction to performance on the job. The results generated by the Record of Management Practice are tabulated in Table 3. Note, it does not represent absolute time in hours or minutes but rather the percentage of the subjects' time spent in each particular role. FAVOURABLE (F) MEAN UNFAVOURABLE (U) MEAN NON-ATTENDANCE (N) MEAN INTERPERSONAL Figurehead 2,46 3,96 8,17 4,81 4,85 1,01 2,14 2,25 2,07 1,87 ,94 1,29 2,6 1,81 1,66 Leader 7,46 6,86 7,58 8,63 7,63 2,16 2,53 2,68 2,8 8,54 ,69 ,26 1,21 ,95 ,78 Liaison 2,3 5,81 8,69 9,56 6,50 3,22 1,59 8,83 10,04 5,92 19,22 19,02 9,25 9,99 14,32 INFORMATION Disseminator 18,09 18,12 19,62 18,28 18,58 34,18 55,2 33,17 10,04 41,35 15,4 23,03 25,82 21,78 21,51 Spokesman 4,99 7,71 8,3 11,92 8,23 4,22 2,58 3,64 2,68 3,28 6,45 3,23 12,02 8,02 7,43 Monitor Solicited 11,32 12,78 11,82 8,98 11,05 19,83 20,31 10,21 16,09 16,61 10,57 19,66 10,44 21,57 15,56 Nominator Unsolicited 15,46 25,13 18,88 12,52 18 9,89 4,82 5,26 6,19 6,54 28,13 11,51 10,56 11,37 15,39 DECISIONAL Disturbance Handler 2,68 3,8 2,13 4,11 3,19 1,49 1,19 5,18 4,61 3,18 1,25 1,19 8,53 6,25 4,31 Entrepreneur 10,25 3,96 7,33 6,01 6,99 1,3 1,24 2,27 1,52 1,58 4,7 2,72 8,62 5,36 5,35 Resource Allocator 4,71 7,02 6,9 4,61 5,81 11,04 9,2 18,93 10,66 12,47 9,89 6,6 10,37 7,58 8,6 Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 8 The chi square test of independence was applied to the mean scores on the following basis: − Favourable versus Unfavourable Assessment Centre score − Favourable versus Non-Attendance − Unfavourable versus Non-Attendance. The results are reflected in Table 4. TABLE 4 THE CHI SQUARE TEST OF INDEPENDENCE OF THE DIFFERENT SAMPLE GROUPS, BASED ON THE DATA GENERATED BY THE RECORD OF MANAGEMENT PRACTICE Sample Group X 2 Critical X 2 Favourable versus Unfavourable 26,82 (,05) = 16,9 Favourable versus Non-Attendance 12,08 (,01) = 21,7 Unfavourable versus Non-Attendance 18,93 (,005) = 23,6 The data in Table 4 indicates that in the case of matching the Favourable Assessment Centre scorers with the Unfavourable Assessment Centre scorers, the two groups are distinctly different in their role sets. This indicates that there is an interrelationship between their Assessment Centre scores and their role set. As regards the relationship between the Favourable Assessment Centre scorers and the Non-Attendants, there is no such distinct relationship (X 2 = 12,08) and it cannot be inferred that those who have been rated favourably perform differently from those who have never attended. Also, the relationship between the Unfavourable Assessment Centre scorer and the Non-Attendant was significant (X 2 = 18,93). This implies that those who were rated unfavourably on the Assessment Centre have exhibited a role set which is distinctly different from those who have never attended an Assessment Centre. These results suggest that those participants in the Assessment Centre who were rated favourably, and those who had never attended an Assessment Centre, actually perform differently on the job to those rated unfavourably. As such, the null hypothesis was rejected, and the alternative hypothesis accepted - namely, that there is a relationship between the role sets of the subjects, and their association with the Experimental and Control Groups. Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 9 In order to determine the statistical significance of the variance of individual role sets across the three sample groups and notwithstanding the small sample sizes, a one-way analysis of variance was conducted for each role set, using the new data in Table 3. This analysis is represented in Table 5. A two-way analysis of variance was conducted for the Monitor Role, treating 'solicited' and 'unsolicited' information as separate entities. An interaction effect between the two was identified with an F score of 15,33. The tabled values were 3,55 and 6,01 respectively for the (,05) and (,01) levels of significance (degrees of freedom 2,18). It will be noted that no mention is made of the Negotiator Role (a decisional role included by Mintzberg) in the results. This is because within the investigators' interpretation of how the roles should be applied to middle management, we felt there was no opportunity to exercise a pure negotiator role. And as the negotiator role is a hybrid role at the best of times, any possible components were justifiably allocated elsewhere. TABLE 5 THE RESULTS OF THE ONE-WAY ANALYSIS OF THE VARIANCE OF THE SCORES WITHIN EACH INDIVIDUAL ROLE-SET Source of Variation M.S. M.S. F (Role) w b df Figurehead 2,23 12,74 9:2 5,71* Leader 0,21 50,97 9:2 242,71** Liaison 19,36 88,26 9:2 4,56* Disseminator 41,35 613,93 9:2 14,85** Spokesman 7,36 28,24 9:2 3,84 Monitor 45,72 66,20 9:2 1,45 Distrubance Handler 6,23 1,79 9:2 0,29 Entrepreneur 4,39 29,80 9:2 6,79* Resource Allocator 8,17 44,73 9:2 5,47* * p < ,05 ** P < ,01 A brief interpretation of these results follows. The analysis of variance revealed a significant difference in the variance of the three groups for all the roles except the Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 10 Disturbance Handler and Spokesman roles. The investigator hypothesized that the reason for this is probably that the subjects of this study, occupying upper-middle management positions, have frequent recourse to company policy and the example of others when handling irregular situations, in addition to being slightly removed from direct contact with the external environment as it exerts pressure on the organization. As such, these two roles do not provide much opportunity within themselves for the subjects of the study to express themselves therein, and therefore no significant variance amongst the groups was detected. It has been shown in the Interpersonal roles that those participants who received favourable ratings on the Assessment Centre, spent more time on the Figurehead role when observed on the job than those who received unfavourable ratings, whilst the Control Group lay in-between. This could possibly be interpreted as a function of the superior ability of the Favourable scorers to organize their affairs efficiently, such that due to adequate planning and control measures their work pressures are streamlined. The Leader role too showed a significant variance between the two Assessment Centre groups. The high scorers are seen to spend relatively more time in this role than the lower scorers, who appear to neglect it somewhat. Contrary to the Figurehead role, in this case the Control Group measure for the Leader role is less than the other two groups - thus indicating the educational and developmental benefits of the Assessment Centre, in that all participants appear to derive some benefits from attendance, which distinguish them from those who have never attended. With respect to the Informational roles it has been shown that a significant difference in variance existed between the groups with regard to the Disseminator and Monitor roles. Once again, this has been interpreted as being due to the superior ability of the Favourable scorers to organize their affairs. In this instance it is felt that due to their superior organizational and management ability the Favourable scorers have less need and less inclination to spend excessive time on the dissemination of information, than the Unfavourable scorers. Expanding on this viewpoint with reference to the Monitor role, with superior planning and organization less time is spent in acquiring information, whilst more unsolicited information flows in, than is the case with the Unfavourable scorers. Finally with regard to the Decisional roles, the significant variance in the Entrepreneur role is attributed to the fact that those who scored higher on the Assessment Centre have Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 11 illustrated a greater awareness of the factors influencing their work and tend to react in a more constructive manner. It is noted that the Control Group scores lie in-between the scores of the two Experimental Groups, thus indicating that the low scores on the Assessment Centre actually have revealed an interest in adjustment and change that is below the norm for the population. Similarly, an examination of the Resource Allocator role data shows the Control Group bisecting the scores of the other two groups, thus indicating that the high scorers and low scorers on the Assessment Centre each occupy opposite extremes from the norm. In this case once again, the organization and efficiency as well as judgement of the high scorers can be referred to as the reason they spend far less time than the low scorers on organizing their affairs and the affairs of their department. It was thus established that Assessment Centre ratings are related to performance on the job. The role sets of those managers who were rated Unfavourably at the Assessment Centre differed significantly from the role sets of those managers who were rated Favourably, and the null hypothesis was rejected. The difference in the role sets of the sample groups was most prominent between the group of managers who had received Favourable ratings, and those who had received Unfavourable ratings. The difference in the role sets between the groups who had not attended an Assessment Centre and those who received Unfavourable ratings was also significant. The particular roles which contributed to this significant variation between the three groups were the following: Figurehead, Leader, Liaison, Disseminator, Entrepreneur, Resource Allocator, Monitor. CONCLUSION The results in Phase 2 implied that the particular Assessment Centre had the potential to classify managers effectively, in a manner that could have a positive impact on the workplace. This is a significant finding in terms of the objective of the organization. Note that at no stage has it been stated that one configuration of roles is superior to another. It has merely been stated that differences have been observed, and that such differences correspond to Assessment Centre ratings. As such, the Assessment Centre investigated in this study generated information which could be utilized by the subject company in the development of a more efficient managerial work force. However, for such development to occur it is essential that the Assessment Centre be regarded as more than a fingertip guide to whom to promote and whom to hold back, and Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 12 more than a one-stage assessment of individuals. In fact, as a pure discriminator it is not a practical tool as it is only effective at the two extremes, i.e. the extremely incompetent and the extremely capable. This is evident in the first phase of this study when a pure classification of the participants on the Assessment Centre could only be established between those who had been rated extremely Favourably and those who had been rated extremely Unfavourably. The performance of a manager, whilst taking into account isolated managerial skills, should be viewed in the context of his peculiar work environment. The Assessment Centre should be seen as a source of reliable input into a total system of organization development. Ideally the organization should be modified to accommodate such a method as the Assessment Centre, which should be used systematically in conjunction with manpower planning, training and performance appraisal, incorporating the appropriate channels of information flow. The Assessment Centre and information generated by it has a valuable role to play in the functioning of an organization, but its contribution should not end at the close of a particu- lar centre. OPSOMMING In hierdie studie is 'n "ex post facto" ondersoek gedoen na die geldigheid van 'n besondere takseersentrum soos dit deur 'n versekeringsmaatskappy geadministreer word. Daar is vas- gestel dat die besondere takseersentrum 'n geldige identifiseerder van bestuurspotensiaal is. Daar is voorgestel dat die organisasie groter voordeel uit die takseersentrum kan put indien dit nie in isolasie beskou word nie, maar tesame met addisionele insette in terme van die huidige funksionering van die organisasie in werking gestel word. Die takseersentrum behoort ook 'n integrale deel van die inligtingsvloei deur die organisasie te vorm. REFERENCES Byham, W. Validity of assessment centres. Symposium presented at the meeting of the American Psychological Association. Washington, 1971. Christie, R. & Creis, F. Studies in Machiavellianism. New York: Academic Press, 1970. Crumbaugh, J .C. Cross validation of the Purpose-in-Life Test based on Frankl's concepts. Journal of Individual Psychology, 1968, 24, 74-81. Crumbaugh, J.C. & Maholick, L.T. An experimental study in existentialism. In V.E. Frankl, Psychotherapy and Existentialism. Harnibdswirth, Pelican Books, 1973. Finkle, R. Managerial assessment centres. In M. Dunnette (ed), Handbook of Industrial and Organizational Psychology. Chicago: Rand-McNally, 1976. Perspectives in Industrial Psychology 1982 8.2 Validity of the Assessment Centre (I.G. Sakinofsky and I. van W. Raubenheimer) 13 Mintzberg, H. The nature of managerial work. New York: Harper and Row, 1973. Moses, J.L. Standards and ethical considerations in assessment centre operations. Task Force on Development of Assessment Centre Standards, Canada, 1975. Stogdill, R.M. Manual for the LBDQ-Form xii - an experimental revision. Columbus: Ohio State, 1963. ___________________