6-VorsterRoodt.qxd On competencies Competencies have wide application and serve as a foundation for various human resource processes, such as recruitment and selection, training and development, performance management, and career and succession planning (Weightman, 1994). Competencies according to McLagan and Nel (1995, p.49) “… are the personal resources that help make a system of governance work”. It is therefore important that the competency approach be optimised. To be competitive in the global market, South Africa has to distribute its available and required skills through the whole population. This need was reflected in South Africa’s position in the World Competitiveness Report of 2001 (World Economic Forum, 2001). In this report, South Africa was placed 34th out of 75 countries, compared to 33rd out of 59 in 2000 (World Economic Forum, 2000) and 47th out of 59 in 1999 (World Economic Forum, 1999) in relation to growth and development. Thus, the assessment, development and acquisition of competence become imperative to drive a country’s growth and development. The Skills Development Act, (No 97 of 1998), and the South African Qualifications Authority Act, (No 58 of 1995), are legislative initiatives focused on addressing the aforementioned challenge. Table 1 summarises the objectives of the Skills Development Act and the South African Qualifications Authority Act. From the summary of objectives in Table 1, it is clear that developing people, as well as giving credit for development, is to be promoted. This brings to mind a question about the relevancy and meaning of competency-based processes in managing human resources. According to Weightman (1994, p. 11) “using competencies as the basis for National qualification, and then equivalences across borders, alleviates the difficulties of differences between job titles and cultures, as competencies concentrate on the ability to do the job”. It is therefore necessary that any competency-based assessment approach for training and development purposes in particular, will have to meet the objectives as described in the said legislation. TABLE 1 OBJECTIVES OF THE SKILLS DEVELOPMENT ACT (NUMBER 97 OF 1998) AND SOUTH AFRICAN QUALIFICATIONS AUTHORITY ACT (NUMBER 85 OF 1995) Skills Development Act South African Qualifications (Number 97 of 1998) Authority Act (Number 58 of 1995) To develop the skills of the South To create a national framework for African workforce the learning environment To increase investment in education, To facilitate access, mobility and training and development and progression within education, return on investment training and career paths To encourage employers to create To promote the quality of education an environment for learning and and training create opportunities to develop skills, gain experience and get employment To encourage employees to learn To redress past unfair discrimination practices To improve employability of To contribute to the full personal previously disadvantaged groups development of each learner and redress past disadvantages To ensure quality of education, training and development To assist work seekers, retrenched workers and employers seeking competent people To provide and regulate employment services With this in mind, two basic issues arise, namely what a competency is and what the relationship between national standards and competencies are. A VORSTER G ROODT Department of Human Resource Management Rand Afrikaans University ABSTRACT The primary purpose of this study was to evaluate the possibility of using a psychometric approach for assessing supervisory competencies relevant to the mining and refining environment. The competency questionnaire was developed using supervisory roles and registered supervisory unit standards from the United Kingdom (UK), as no registered unit standards exist in South Africa. Twenty-four supervisors from three departments (Production, Engineering and Laboratory) were evaluated by 125 raters; besides by themselves, also by their managers, peers, customers and their sub-ordinates. Based on difference scores derived from the Importance and Performance scales, a single factor was extracted with an internal reliability of 0,965. No statistical significant differences were obtained (ANOVA’s, t-test and F-statistics) between groups based on biographical variables or between rater groups. The findings and their implications are further discussed. OPSOMMING Die primêre doel van die studie was om die moontlikheid vir die gebruik van ’n psigometriese benadering tot toesighouerbevoegdheidsbeoordeling, te evalueer. Die bevoegdheidsvraelys is ontwikkel deur gebruik te maak van toesighouersrolle en geregistreerde toesighouerseenheidstandaarde van die Verenigde Koningkryk, as gevolg van ‘n gebrek aan bestaande eenheidstandaarde in Suid-Afrika. Vier-en-twintig toesighouers van drie departemente (Produksie, Ingenieurswese en Laboratorium) is deur 125 beoordelaars geëvalueer; buiten deur hulself, ook deur hul bestuurders, kollegas, kliënte en hul ondergeskiktes. ’n Enkele faktor, met ’n betroubaarheid van 0,965, gebaseer op die verskiltellings van die Prestasie- en Belangrikheidskaal, is onttrek. Geen beduidende verskille (ANOVA’s, t-toetse en F-statistiek) kon tussen groepe gebaseer op biografiese veranderlikes en die onderskeie beoordelaarsgroepe gevind word nie. Hierdie bevindinge en die implikasies daarvan word verder bespreek. A PSYCHOMETRIC APPROACH TO SUPERVISORY COMPETENCY ASSESSMENT Requests for copies should be addressed to: A Vorster, Department of Human Resource Management, RAU University, PO Box 524, Auckland Park, 2006 38 SA Journal of Industrial Psychology, 2003, 29 (3), 38-44 SA Tydskrif vir Bedryfsielkunde, 2003, 29 (3), 38-44 The world of vocational education and training uses two distinct categories of competency models, namely 1) Competency models that are inputs based, and 2) Competency models that are outcomes based (Buckley & Caple, 1995; Mansfield, 1989). The first category constitutes descriptions of individual knowledge and skills. It is based on the assumption that competencies are individual attributes. The second category is based on descriptions of work role outcomes. For the purpose of this study, competencies are defined as “skills or clusters of skills performed in a particular context or environment to specific standards” (Bellis, 1999, p. 19). This definition supports the notion of competencies being outcomes based, thus establishing a strong relationship with national standards, which are also outcomes based (Hallendorff, Richardson & Wood, 2000). It is therefore proposed that competencies as standards, and national unit standards should be seen as being the same. The underlying principle of this competency approach offers the organisation access to a framework (i.e. National Qualifications Framework) of assessing and developing staff, which is formal, structured, reliable and recognised at national and industry level. The individual benefits from this competency approach through its transferable accreditation of competencies acquired. To be useful in any applied assessment context, competency standards have to be defined, described and assessed in terms of the behaviour they consist of. This will constitute the content validity of the assessment. It should thus be done in such a way that it accommodates most or all of the uses of competency standards, as mentioned before. Buckley and Caple (1995) and Mitchell (1989) identified the components of a competency standard as performance criteria, range statements, related knowledge and assessment requirements, which are similar to the key components thereof as described by Hallendorff et al (2000). These key components are: competency standard title (an outcome large enough to be recognised and credited), specific outcomes (which are smaller outcomes of the unit standard title), assessment criteria (which explains how well the specific outcomes should be performed) and range statements (used to describe the context or environment). These components underpin an approach of providing evidence as proof of competency during the standard-based assessment process. In the vocational training process, specified learning outcomes that cover the aims of the curriculum, could be taught and assessed in unambiguous ways, which would overcome the problems of reliabilit y, content validit y and fairness in assessments. The objective therefore, is to base learning on outcomes-based competency standards (Gonczi, 1999). It would therefore be beneficial for organisations to apply the outcomes-based assessment and development of, specifically, key employees such as supervisors, cost-effectively. The roles of supervisors will therefore be discussed in more detail below. On the role of supervision Managers, as supervisors do, have many roles to play in the organisation. These roles are: � Figurehead (performs ceremonial and symbolic duties such as greeting visitors or signing legal documents); � Leader (direct and motivate subordinates, communicating with subordinates, training and counseling them); � Liaison (maintains information links both inside and outside organisation); � Monitor (seeks and receives information, scans reports and maintains personal contacts); � Disseminator (forward information to other organisation members, e.g. by sending memos and reports); � Spokesperson (transmits information to outsiders through speeches, reports and memos); � Entrepreneur (initiates improvement projects and identifies new ideas); � Disturbance Handler (takes corrective action during disputes or crises and resolves conflicts among subordinates); � Resource Allocator (decides who gets resources through scheduling, budgeting and setting priorities); � Negotiator (represents department’s interests in meetings and negotiations). The functions that managers fulfil, are contained in their roles. This theory brings a different dimension to analysing management through focusing on the role instead of the function. The first line supervisor performs all the managerial roles of top management, but it is done within the context of his or her own section (Mintzberg, 1973). Organisational co-ordinating mechanisms in this supervisory context are: “the glue holding organisational structure together” (Mintzberg, 1979, pp. 3-7). These mechanisms involve: i) mutual adjustment, ii) direct supervision, (own emphasis), iii)standardisation of work processes, iv) standardisation of skills, and v) standardisation of outputs/results. As organisational work becomes more complex, co-ordination seems to shift from mutual adjustment to direct supervision, to standardisation and can finally revert back to mutual adjustment. He postulated that: “direct super vision achieves co- ordination by having one individual taking responsibility for the work of others” (Mintzberg, 1979, pp. 3-4). This is evident with large groups that are less able to co-ordinate informally, thus with further complexity, supervision becomes necessary. Of these five elements, direct supervision and standardisation of skills support the main focus of the current study. The necessity of supervision in the modern organisation can be attributed to new technology and a change in worker values (Hodgetts, 1987). Direct supervision, according to Mintzberg (1979), included planning, scheduling, allocating, instructing and monitoring actions, and closely resembles what Hilgert and Leonard (1998,) (planning, organizing, staffing, leading and controlling), as well as Hodgetts (1987) (planning, organising, directing and controlling) described as supervisory functions. These supervisory functions could be grouped in the same groupings as Mintzberg (1973) has done with managerial activities and roles as illustrated in Table 2. The supervisory competencies, according to UK standards, are also listed in the last column to indicate the overlap with supervisory functions and roles. TABLE 2 COMPARISON OF MINTZBERG’S MANAGERIAL ROLES, SUPERVISORY FUNCTIONS AND THE COMPETENCIES EXTRACTED FROM UK UNIT STANDARDS Managerial Associated roles Supervisory Supervisory Activities Functions competencies Interpersonal � Figurehead � Instructing � Building teams Relationships � Liaison � Communicating � Leader � Acting assertively � Influencing others Information � Monitor � Monitoring � Focussing on processing � Disseminator results � Spokesman � Searching for information Making � Improver/ � Allocating � Thinking and significant changer � Planning taking decisions decisions � Disturbance � Scheduling � Behaving handler ethically � Resource � Managing self allocator � Negotiator SUPERVISORY COMPETENCY ASSESSMENT 39 Companies are often encountering difficulties in the area of supervision. These difficulties refer to problems with communications, controlling internal mobility, managing change, managing subordinates, performance management and measurement, and managing discipline and grievances (Calhoon, 1971). These difficulties can be related to the roles of supervisors as was described above. It is argued that the problems as mentioned, can be addressed through ensuring the competence (standardisation of skills) of supervisors in the roles that they have to play. Multi-rater performance evaluations play a valuable role in this regard. On 360° competency assessments As mentioned before, there are different approaches towards assessing competencies. Standards-based competency assessment can be a long and tedious process due to its nature of seeking proof or evidence of competence. It is highly likely that this process will be biased when performed by a single individual. Different sources of bias in performance ratings were identified by Theron and Roodt (1999, 2000) who listed different working relationships; ego-centric bias; differences in organisational level; rater-leniency; the halo effect; purpose of ratings; and differences in rating criteria as examples. Theron and Roodt (2000) also added different mental models of raters to this list, based on their own research. Mental models in this instance refer to “unitary, spatial models where distance has functional consequence, and which are used to give meaning and understanding to complex systems or phenomena” (Theron & Roodt, 2000, p. 15). Quantitative evidence of the validity and reliability of such single-sourced, standards-based competency assessments are therefore difficult to determine. Following a multi-rater approach on the other hand, such as 360° competency assessments, may address most of these concerns by generating quantitative, psychometric information on reliability and validity coefficients. In 360° competency assessments, according to Jones and Bearly (1996), an individual receives feedback from a multi-rater assessment process, providing a more objective, holistic view of the individual. Overall, 360° assessments are more accurate, credible, fair and motivational than single-source assessments (Edwards, 1998) when applied effectively. The success of 360° assessments (multi-rater assessment) is largely dependent on effective integration with human resource systems and business strategy, valid content, accurate responses, clear and specific results, integration with career development and planning support, and measurable performance improvement (Morical, 1999). In order to achieve this, specific attention should be given to design, implementation and evaluation factors (Theron & Roodt, 2001). More specifically, users of the system should be allowed to participate in the development of the system, to ensure that the system will be perceived as fair. Users of the system should also be trained in the proper use of the system (Edwards & Ewen, 1996), and the requirements of the Employment Equity Act (No 55 of 1998) that requires all psychological tests and other similar assessments to be valid and reliable, fair as well as not biased against any employee or any group of employees, should be adhered to. This necessitates that any 360° assessment rating system should be evaluated and validated. In this study, the 360° competency assessment approach is seen as supplementary to the formal standards-based competency approach, as it is based on registered UK standards. The statistical reliability and validity of the 360° competency assessment process can be defended and could possibly be included in the standards-based assessment process. In view of the discussion in the section above, the research objectives are stated as follows: Research Objectives The primary objective of the study was to determine whether a 360° degree competency assessment questionnaire could reliably and validly assess supervisory competencies. The secondary objectives of the study were: � To determine if there are any statistically significant differences in the mean difference scores between the Importance and Performance scales, with regard to different work departments, educational qualification levels and language groups. This comparison would assess whether different comparison groups have different mental models regarding supervisory competencies. � To determine if there are any statistically significant differences in the mean difference scores between the Importance and Performance scales, with regard to different rater groups (manager, self, customer, sub-ordinate and peer/colleague). This comparison would assess possible differences in mental models on supervisory competencies between different rater groups. METHOD The research participants The study was conducted in the mining and refining industry during 2002. A group of twenty-four first line supervisors from three departments (Production, Laboratory and Engineering) was assessed using the 360° Supervisory Competency Inventory (360° SCI). The total population of participants (N = 125) in this study were from 5 different rater groups: managers, self, customers, peers/colleagues, and subordinates. Participating employees were briefed about how to complete the rating form, the aim of the assessment and the purpose of 360° assessment. Table 3 provides a brief overview of the biographical background of the rater sample in this study. TABLE 3 DISTRIBUTION OF PARTICIPANTS ACCORDING TO BIOGRAPHICAL VARIABLES Demographic variable 1. Department N % Production 65 52 Laboratory 38 30,4 Engineering 22 17,6 Total 125 100 2. Relationship N % Self 24 19,2 Manager 40 32 Customer 11 8 Subordinate 38 30,4 Peer/colleague 12 9,6 Total 125 100 3. Qualification N % Std 8 – 10 77 61,6 Tertiary 44 35,2 Missing 4 3,2 Total 125 100 4. Language N % Afrikaans 43 33,3 English 22 17,1 Zulu 34 26,4 Other 30 23,3 Total 129 100 5. Gender N % Male 119 95,2 Female 6 4,8 Total 125 100 VORSTER, ROODT40 The level of education of the participants ranges from standard 8 to tertiary qualifications. Four language groupings were included, namely Afrikaans, English, Zulu and Other. It should be noted that 129 respondents (more than 125) are listed under language groups, which can be attributed to multi-lingual individuals who listed more than one home language. The measuring instrument Accepted United Kingdom standards for management and supervision were analysed and compared with in-house supervisory roles. The obtained results were further validated by subject matter experts, such as line management and individuals drafting standards for the Standards Generating Body for management and supervision. The nine supervisory competency dimensions (refer to Table 2) extracted from the unit standards were: Team building; Communicating; Focusing on results; Thinking and taking decisions; Acting assertively; Managing self; Influencing others; Searching for information; and Behaving ethically. Statements were developed around these nine competency dimensions for the 360° SCI. The 360° SCI included a total of 34 behaviourally based statements. Examples of items used were as follows: “The person I am assessing: – Motivates team members to do a good job. – Seeks to clarify conflicting information.” Participants responded to the items of the 360° SCI on two scales, namely a six-point scale in respect of actual Performance and a six-point scale in respect of Importance (in terms of overall performance), where a 1 on the response scale was ‘strongly disagree’ (meaning poor performance), and a 6 ‘strongly agree’ (meaning good performance) or ‘low importance’ and ‘high importance’ respectively. The research procedure The Business Manager of the Business Unit provided a letter of support to the project to explain the objective of the project and to urge participants to be honest and co-operative. This letter was attached to each form (the assessment was paper- based) and distributed to the participating employees with instructions on how to complete the 360° SCI and whom they were to assess. A deadline date for submitting the completed questionnaire was also communicated. Senior staff members from each section or department were identified as contact people for the study, overseeing the distribution, completion, collection and control of the 360° SCI forms. This ensured the integrity of the assessment tool and the data collected for the study. RESULTS First level factor analysis on the item inter-correlation matrix The initial 34 items were inter-correlated. Owing to a lack of space, the inter-correlation matrix cannot be reproduced here. A Kaiser-Meyer-Olkin measure of sampling adequacy (MSA) of 0,906 was obtained which indicated that the unreduced matrix was suitable for a first level factor analysis. Item 23 was omitted from the inter-correlation at this point. Eigenvalues were calculated on the inter-correlation of the remaining 33 items and six factors were postulated according to Kaiser’s (1961) criterion (eigenvalues larger than unity). These six factors explained 71,738% of the variance in the factor space, as reflected in Table 4. Six factors were extracted using principal axis factoring. Meaningful item loadings were obtained on all six postulated factors. Subsequently, the sub-scores were calculated for these six factors. TABLE 4 EIGENVALUES OF THE UNREDUCED INTERCORRELATION MATRIX OF THE 360° SCI Initial Eigenvalues Root Eigenvalues % of Variance Cumulative % 1 15,937 48,293 48,293 2 2,386 7,230 55,523 3 1,671 5,064 60,587 4 1,476 4,474 65,061 5 1,153 3,495 68,556 6 1,050 3,182 71,738 7 0,927 2,809 74,546 8 0,838 2,538 77,084 9 0,754 2,286 79,370 10 0,726 2,201 81,572 11 0,668 2,025 83,597 12 0,576 1,746 85,343 13 0,524 1,589 86,932 14 0,512 1,551 88,482 15 0,469 1,422 89,904 16 0,392 1,189 91,093 17 0,333 1,010 92,103 18 0,318 0,962 93,066 19 0,304 0,921 93,986 20 0,251 0,761 94,748 21 0,233 0,707 95,455 22 0,204 0,619 96,074 23 0,194 0,589 96,663 24 0,178 0,540 97,203 25 0,147 0,445 97,649 26 0,140 0,424 98,072 27 0,124 0,377 98,449 28 0,121 0,365 98,815 29 0,108 0,327 99,142 30 0,087 0,265 99,407 31 0,075 0,228 99,635 32 0,070 0,212 99,847 33 0,051 0,153 100,000 Trace 33,000 Item 23 was omitted before the intercorrelation was performed Second level factor analysis on the sub-score inter- correlation matrix The six sub-scores (obtained from the first level factor analysis) were inter-correlated and the results of the inter-correlation of the sub-scores are displayed in Table 5. A Kaiser-Meyer-Olkin measure of sampling adequacy of 0,904 was obtained on this inter-correlation matrix, which indicated that this matrix was also suitable for the second level factor analysis. TABLE 5 INTERCORRELATION MATRIX OF THE SUBSCORES OF THE 360° SCI Factor 1 2 3 4 5 6 1 1 0,785 0,478 0,737 0,800 0,728 2 0,785 1 0,594 0,730 0,739 0,741 3 0,478 0,594 1 0,487 0,430 0,560 4 0,737 0,730 0,487 1 0,702 0,626 5 0,800 0,739 0,430 0,702 1 0,658 6 0,728 0,741 0,560 0,626 0,658 1 All correlations are significant at the 0,01 level (2-tailed) SUPERVISORY COMPETENCY ASSESSMENT 41 Table 6 provides the eigenvalues of the unreduced inter- correlation matrix of sub-scores. A single factor was postulated based on Kaiser’s (1961) criterion. This factor explained 71,606% of the variance of the factor space. TABLE 6 EIGENVALUES OF UNREDUCED INTERCORRELATION MATRIX (6×6) Initial Eigenvalues Root Eigenvalues % of Variance Cumulative % 1 4,296 71,606 71,606 2 0,660 10,998 82,604 3 0,364 6,061 88,664 4 0,283 4,722 93,386 5 0,214 3,563 96,949 6 0,183 3,051 100,000 Trace 6,000 The factor solution converged after six iterations. As only one factor was extracted, no further rotation took place. Table 7 provides the factor matrix of the 360° SCI. TABLE 7 FACTOR LOADINGS OF THE 360° SCI Subscores Items N Factor 1 h 2 j 1 3,4,18,1,2,28,7,19 8 0,894 0,799 2 14,21,12,24,17,10,26,11,13 9 0,904 0,817 3 25,34,22,30,20 5 0,599 0,358 4 8,6,29,33,27 5 0,812 0,659 5 9,15,5 3 0,833 0,694 6 32,31,16 3 0,815 0,665 variance explained of factor space = 71,606% Cronbach alpha = 0,965 Iterative item analyses were performed in respect of the single factor for both the Performance and the Importance measures of the 360° SCI and high Cronbach alpha coefficients of 0,968 and 0,964 respectively were obtained. The results will not be presented separately for each scale, owing to limited space. Difference scores were used to present the scale properties. A high Cronbach alpha coefficient of 0,965 was obtained in respect of the difference scores of these two scales. Table 8 provides the item statistics of the difference scores of the two scales in respect of the 360° SCI. It is clear from Table 8 that the corrected item-total reliabilities vary between 0,330 and 0,815 with only five items having values below 0,50 that contributed to the high internal consistency of the scale. The above results suggest that supervisory competencies could be reliably (high Cronbach alpha coefficient of 0,965) and validly [based on factorial validity – a component of construct validity (Allen & Yen, 1979)] measured which support the primary objective of this study. The first secondary objective of the study was to determine whether there were any statistically significant differences in the mean difference of competency scores of groups created in terms of the different biographical variables. ANOVA’S (more than two groups), t-tests (two groups only) and F-statistics were used for these analyses. As a large majority of the respondents were male and very few females participated in the study, gender was not included in this analysis. TABLE 8 ITEM STATISTICS IN RESPECT OF THE DIFFERENCE SCORES OF THE TWO SCALES OF THE 360° SCI (N = 125) Item Scale Mean Scale Item SD Corrected Alpha if item if item variance if Item-total deleted deleted item deleted correlation V1 26,542 1032,704 32,136 0,773 0,964 V2 26,592 1034,260 32,160 0,761 0,964 V3 26,708 1040,797 32,261 0,692 0,964 V4 26,692 1049,929 32,403 0,675 0,964 V5 26,358 1034,534 32,164 0,714 0,964 V6 26,483 1031,849 32,122 0,712 0,964 V7 26,667 1042,006 32,280 0,667 0,964 V8 26,883 1066,507 32,657 0,487 0,965 V9 26,458 1035,259 32,175 0,767 0,964 V10 26,350 1034,532 32,164 0,753 0,964 V11 26,650 1038,145 32,220 0,738 0,964 V12 26,542 1037,292 32,207 0,760 0,964 V13 26,633 1025,512 32,024 0,795 0,964 V14 26,675 1054,020 32,466 0,551 0,965 V15 26,183 1024,538 32,008 0,799 0,964 V16 26,458 1040,351 32,254 0,739 0,964 V17 26,508 1020,336 31,943 0,815 0,963 V18 26,600 1037,133 32,205 0,772 0,964 V19 26,842 1072,050 32,742 0,484 0,965 V20 27,317 1079,865 32,861 0,364 0,966 V21 26,733 1051,912 32,433 0,621 0,965 V22 27,142 1080,576 32,872 0,330 0,966 V24 27,067 1054,197 32,468 0,587 0,965 V25 26,983 1063,580 32,613 0,522 0,965 V26 26,283 1035,264 32,176 0,702 0,964 V27 26,525 1030,806 32,106 0,697 0,964 V28 26,667 1044,325 32,316 0,704 0,964 V29 26,758 1053,748 32,461 0,596 0,965 V30 26,858 1074,190 32,775 0,481 0,965 V31 26,675 1043,868 32,309 0,705 0,964 V32 26,733 1046,433 32,349 0,708 0,964 V33 26,475 1029,747 32,090 0,729 0,964 V34 26,958 1065,116 32,636 0,624 0,965 No statistically significant differences were found in respect of the mean scores of different departments, qualification levels or home languages. Tables 9, 10 and 11 reflect the results in respect of the mentioned biographical variables. TABLE 9 ANOVA: COMPARISON OF THE MEAN DIFFERENCE SCORES OF THE VARIOUS WORK DEPARTMENTS N Mean SD Std. Error Production 65 0,90 0,968 0,120 Engineering 38 0,84 1,213 0,197 Laboratory 22 0,66 0,774 0,165 Total 125 0,84 1,016 0,091 Sum of DF Mean F-ratio P(F) Squares Square Between Groups 0,890 2 0,445 0,427 0,653 Within Groups 127,066 122 1,042 Total 127,956 124 VORSTER, ROODT42 As the obtained value of F = 0,427 does not exceed F(p=0,05; df=2, 122) = 3,07 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the departments. TABLE 10 T-TEST: INDEPENDENT COMPARISON OF THE MEAN DIFFERENCE SCORES OF THE EDUCATIONAL GROUPS Group Statistics QUALIFICATION N Mean SD Std. Error Mean Std 8-10 77 0,86 0,981 0,112 Tertiary 44 0,85 1,129 0,170 Levene’s Test for t-test for Equality Equality of Variances of Means F-ratio P(F) t-value DF P(t) Mean Difference Equal variances 0,047 0,829 0,061 119 0,952 0,01 assumed Equal variances 0,059 79,674 0,953 0,01 not assumed As the obtained value of t = 0,061 does not exceed t(p=0,05; df=119) = 1,960 and the obtained value of F = 0,047 does not exceed F(p=0,05; df=2, 119) = 3,07 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the educational qualification levels of the sample. TABLE 11 ANOVA: COMPARISON OF THE MEAN DIFFERENCE SCORES OF THE VARIOUS LANGUAGE GROUPS N Mean SD Std. Error English 22 0,93 1,526 0,325 Zulu 34 0,68 0,752 0,129 Afrikaans 42 0,72 0,747 0,115 Other African Languages 29 1,23 1,128 0,210 Total 127 0,86 1,024 0,091 Sum of DF Mean F-ratio P(F) Squares Square Between Groups 6,115 3 2,038 1,988 0,119 Within Groups 126,119 123 1,025 Total 132,234 126 As the obtained value of F = 1,988 does not exceed F(p=0,05; df=3, 123) = 2,68 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the different language groups. The last secondary objective of the study was to determine if there were any significant differences bet ween the mean difference scores of the different rater groups. ANOVA’s and F-statistics yielded the following results in Table 12: As the obtained value of F = 1,311 does not exceed F(p=0,05; df=4, 120) = 2,45 it is concluded that there are no statistically significant differences in respect of the mean difference scores of the rater groups. This finding supports the last of the secondary objectives. TABLE 12 ANOVA: COMPARISON OF THE MEAN DIFFERENCE SCORES OF THE VARIOUS RATER GROUPS N Mean SD Std. Error Manager 24 0,84 1,045 0,213 Self 40 0,97 1,204 0,190 Subordinate 38 0,70 0,889 0,144 Customer/Supplier 11 0,45 0,308 0,093 Peer/Colleague 12 1,29 1,042 0,301 Total 125 0,85 1,021 0,091 Sum of DF Mean F-ratio P(F) Squares Square Between Groups 5,412 4 1,353 1,311 0,270 Within Groups 123,792 120 1,032 Total 129,204 124 DISCUSSION The primary purpose of this study was to evaluate the possibility of using a psychometric approach for assessing supervisory competencies relevant to the mining and refining environment. The results obtained from the factor analyses and the item analyses indicate that the construction of the 360° SCI was based on sound psychometric principles. The factor analysis resulted in a single factor that indicates a solid theoretical base and sound procedure in the construction of the 360° SCI. A high level of reliability and a limited amount of error variance in the measurement of the construct of “supervisory competency” was obtained, indicated by the high internal consistency of 0,968 and 0,964 for the Performance and Importance scales respectively. The difference scores between these two scales yielded an Alpha coefficient of 0,965. These results indicate that supervisory competencies can be assessed in a reliable and consistent manner with the 360° SCI. The content validity (based on the content of unit standards) supports the face validity of the instrument that was based on the comparison of supervisory competencies and Mintzberg’s managerial roles (1973) as indicated in Table 2. It is clear that the 360° SCI can be used for supervisors in different functional contexts within in the mining and refining industry, as no significant differences were found between the mean difference scores for the three departments that participated in this study. The 360° SCI also seems to be fair and equitable, as no significant differences were found in the mean difference scores with regard to the qualification level or language groups. As only six females participated in the study, gender groups were not included in the analysis. It appears that the 360° SCI is not biased in terms of the biographical differences analysed, which supports the first of the secondary objectives of this study. No significant differences were obtained on the ratings of the different rater groups (self, manager, peer, customer/supplier and sub-ordinate). This is often not the case for first time (or even second time) ratings as pointed out by Theron and Roodt (1999) and Theron and Roodt (2000). This implies that the different rater groups often apply different mental models for interpreting the questionnaire, resulting in significantly different ratings. SUPERVISORY COMPETENCY ASSESSMENT 43 The implications of these findings are far-reaching for organisations, as well as at a national level. The 360° SCI provides a cost-efficient and time-efficient way of assessing the supervisory competencies of individuals in organisations. The time consuming process of declaring a person competent to supervise people through evidence-seeking, standards-based assessments, can be made more effective and efficient. The 360° assessment approach has added advantage in that it seems to be a valid and reliable way of measuring supervisory competencies. One of the shortcomings of the study was that there were very few female participants in the study. Another shortcoming was that the rater sample was relatively small, even though sufficient sampling adequacy was achieved. In future studies it might be useful to consider the following: � Including more female participants in all the rater groups. The results may be very different when this level of diversity is introduced. � Considering the mental models that the different rater groups use as frames of reference, even though no significant differences were found between the rater groups in this study. � Researching the possibilities of using the instrument for developmental and promotional purposes or remuneration increases. � Investigating the possibility of measuring other competencies with a multi-rater assessment approach. � Testing the research against current legislation and practical or operational requirements. � Using a larger sample of raters and ratees. � Including other units within the organisation in the study to test its applicability, reliability and validity for the larger organisation and not just one unit. � Correlating the ratings on the 360° SCI with ratings obtained from supervisory unit standards assessment. The study was aimed at evaluating the possibility of using a psychometric approach for assessing supervisory competencies. The results reported in this article support the use of psychometric assessment (in the form of a 360° supervisory competency questionnaire) in this regard and could be considered as a sound alternative to single-sourced, standards- based competency assessment. REFERENCES Allen, M.J. & Yen, W.M. (1979). Introduction to Measurement Theory. Monterey, CA.: Brooks/Cole Publishing Company. Bellis, I. (1999). Between the rock and the hard place. People Dynamics, (September), 14-19. Buckley, R. & Caple, J. (1995). The theory and practice of training (3rd ed.). London: Kogan Page. Calhoon, R.P. (1971). Cases in personnel management and super vision (2nd ed.). Englewood Cliffs, NJ.: Prentice-Hall. Cannon-Bowers, J.A. & Salas, E. (1997). Team work competencies: The interaction of team member knowledge, skill and attitudes. In H.F. O’Neil (Jr.) (Ed.). Workforce readiness: Competencies and assessment. (pp.151-175) Mahwah, JN: Lawrence Erlbaum. Edwards, M.R. (1998). The future of 360° technologies: Power tools to accelerate workplace learning. Paper presented at the ASTD International Conference, 3 June, San Francisco, CA. Edwards, M.R. & Ewen, A.J. (1996). 360° Feedback. New York: American Management Association. Gonczi, A. (1999). Competency-based learning: a dubious past – a assured future? In D. Boud & J. Garrick (Eds.). Understanding learning at work. (pp.180-195). London: Routledge. Hallendorff, E., Richardson, B. & Wood, B. (2000). Standards writer course. (3rd. ed.). Learning Network SA. Hilgert, R.L. & Leonard, E.C. (1998). Super vision: concepts and practices of management. Cincinnati, Ohio: South Western College. Hodgetts, R.M. (1987). Effective super vision: a practical approach. New York: McGraw-Hill. Jones, E.J. & Bearly, W.L. (1996). 360° feedback: strategies, tactics and techniques for developing leaders. Amherts, MA.:MRD. Kaiser, H.F. (1961). A note on Guttman’s lower bound for the number of common factors. British Journal of Statistical Psychology, 14 (1),1. Mansfield, B. (1989). Competence and standards. In J.W. Burke (Ed.). Competency based education and training. (pp.26-38). London: Falmer. McGuigan, F.J. (1997). Experimental psychology: Methods of research (7th ed.). Upper Saddle River, NJ: Prentice Hall. McLagan, P. & Nel, C. (1995). The age of participation. San Francisco: Berett-Koehler. Messick, S. (1989). Validity. In R.L. Linn. (Ed.). Educational measurement. (3rd ed.). (pp.13-103). New York: American Council on Education & McMillan. Mintzberg, H. (1973). The nature of managerial work. New York: Harper & Row. Mintzberg, H. (1979). The structuring of organisations. Englewood Cliffs, NJ: Prentice-Hall. Mitchell, L. (1989). The definition of standards and their assessment. In J.W. Burke (Ed.). Competency based education and training. (pp.54-64). London: Falmer. Morical, K.E. (1999). A product review: 360° assessments. Training and Development, 53 (4), 43-47. Skills Development Act (no. 97 of 1998). Republic of South Africa. (http://www/labour.gov.za/docs/legislation/skills/act98- 097.html) South African Qualifications Authority Act (no. 58 of 1995). Republic of South Africa. (http://www.saqa.org.za/docs/actd8- 95.html) Theron, D. & Roodt, G. (1999). Variability in multi-rater competency assessment. Journal of Industrial Psychology, 25 (2), 21-27. Theron, D. & Roodt, G. (2000). Mental models as moderating variable in 360-degree competency assessments. Journal of Industrial Psychology, 26 (2), 14-19. Theron, D. & Roodt, G. (2001). An evaluation of the 360° project management competency assessment questionnaire. Journal of Industrial Psychology, 27 (2), 51-56. Weightman, J. (1994). Competencies in action. London: IPD House. Woodruffe, C. (1993). What is meant by a competency? Leadership and Organisation Development Journal, 14 (1), 29- 36. World Economic Forum (1999). World Competitiveness Report. (http://www.weforum.org/site/homepublic.nsf/Content/Glo bal+Competitiveness+Programme). World Economic Forum (2000). World Competitiveness Report. (http://www.weforum.org/site/homepublic.nsf/Content/Glo bal+Competitiveness+Programme). World Economic Forum (2001). World Competitiveness Report. (http://www.weforum.org/site/homepublic.nsf/Content/Glo bal+Competitiveness+Programme). VORSTER, ROODT44