Microsoft Word - 05_Jonny_AK_rev2 Improving the Average Session …… (Jonny) 125  IMPROVING THE AVERAGE SESSION EVALUATION SCORE OF SUPERVISORY PROGRAM BY USING PDCA CYCLE AT PT XYZ Jonny Accounting Department, Faculty of Economic and Comunication, Bina Nusantara University Jln. K.H. Syahdan No 9, Palmerah, Jakarta Barat, 11480 jonny@binus.ac.id ABSTRACT PT XYZ took People Development tasks as important things in order to provide great leaders for handling its business operations. It had several leadership programs such as basic management program, supervisory program, managerial program, senior management program, general management program, and the executive program. For basic management and supervisory programs, PT XYZ had appointed ABC division to solely handled them, while the rest, ABC division should cooperate with other training providers who were reputable in leadership ones. The aim of this study was to ensure that the appropriate leadership style has been delivered accordingly to the guideline to the employees by ABC division to improve the average session evaluation score of the supervisory program by using PDCA (Plan, Do, Check, and Action) cycle. The method of this research was by gathering quantitative and qualitative data by using session and program evaluation format to see current condition. The research finds that the reasons why the program is below target 4,10 score. It is related to the new facilitator, no framework, and teaching aids. Keywords: Session Evaluation, Program Quality Index, Subject Content Index, Facilitator Evaluation Index, Training Effectiveness, PDCA INTRODUCTION As one of world development indicators, market capitalization is very common among investor to evaluate whether a certain nation is attractive or not to them for investing their idle funds in that nation. In 2012, the market capitalization of Indonesia amounted to US$ 428,2 billion (The World Bank, 2016) and it has been reported that 43,72% of this number was derived from top ten companies (Sutyanto, 2013). One of these companies is PT XYZ which is also interesting to be observed on Human Capital Management perspective in order to reveal how it handle its people so that it could derive such astonishing performance among Indonesian companies. This company, naming PT XYZ, is one of the most reputable companies in Indonesia. It has believed that people are one of its important parts of its business sustainability growth. Therefore, it has put people as part of its strategies in order to win the market and develop its competitive advantage aside of its competitors. It is why this company has known for its commitment on people side and won several awards regarding human capital management. Furthermore, it has put people on its mission statement to ensure that its management has committed to people development when developing and nurturing its business. Regarding people development, the company has long realized that there are two functions of its people competencies that need to be nurtured, they are technical and leadership skill. For its technical skill development, the company has decided to delegate this responsibility to its business function with basic thinking that only business function knows what it takes to develop its required 126   Journal The WINNERS, Vol. 17 No. 2, September 2016: 125-132  people competencies. While for leadership skill, the company still believes that it should be handled by its self at head office. This policy is based on Quality Management principle, among these principles is leadership (Hessen, 2015). This principle states that leaders should be able to set a goal and mobilize its people towards it. According to Kaplan (2004), it has stated that it is important to ensure the availability of qualified leaders at all levels to mobilize the organizations towards their strategies. Based on this guideline, the company has further appointed ABC division, which division’s roles are to develop the company’s people and to handle the company’s all level development programs, such as basic management and supervisory programs. While the rest programs are consisting managerial program, senior management program, general management program, and executive program are to be handled by cooperating with training providers. This is done because the company needs to make sure that its people pose the same leadership style when handling its business operation. Therefore, it has set up leadership competency guideline for ABC division to conduct its program in order to ensure that the appropriate leadership style has been delivered accordingly to this guideline. In conducting this supervisory program, ABC division usually asks the favor from other affiliated training centers to provide appropriate facilitators to deliver several sessions during the supervisory program. After the session, participants who are from all affiliated companies are asked to evaluate how well the facilitator has delivered the sessions (FEI). Meanwhile, for each of its session, ABC division takes full responsibility to construct and develop it into the program based on the previous mentioned leadership competency that is required when the participant is about to be promoted to the certain supervisory position in its related affiliated company. These sessions are also evaluated by participants to see how well the sessions meet their needs to acquire leadership competency (SQI). Last but not least, the whole program is also evaluated to see how well the program served the participant needs in acquiring leadership competency so that he/she can boost his/her career (PQI). This is very important in order to see whether the program has been effectively conducted and suitable for its participant in terms of training satisfaction and transfer. Those mentioned indexes are the indicators for the company to ensure that the reaction of the participants is favorable in order to be able to pursue the next steps as proposed by Kirkpatrik (2009), such as learning, behavior, and results. Therefore, the company has set the target of 4.10 as the favorable indicator to reveal that participants give the positive reaction to the program. However, up to batch 46, the average scores for PQI, SCI, and FEI are below targeted 4.10 which were 4,19, 3,95, 3,93 respectively for the last batch of 46. Of course, this condition is not acceptable because these indicators have shown that the program is not effective yet. Therefore, it has triggered with research gap which then becomes research question which is about why the average session scores (PQI, SCI, and FEI) are below required 4,10 as an indicator of its effectiveness and how to improve them. To address this problem, ABC division Head has formed a team to improve this condition by conducting PDCA cycle as its accepted methodology in the team so that those indicators can be above the required 4,10. The result then is important for the division in order to pay its accountability not only for its top management but also its affiliated companies’ management in order to provide reliable leaders which have been trusted to this division to conduct their development issues. Based on Belker et al., (2012), leadership is about how a person sets its goal and leads its people towards the goal. This skill can be born or learned, therefore, it is possible to make an intervention on people to improve their leadership skills. According to this belief, PT XYZ has set its guideline to nurture its people in order to have the appropriate leadership skills required by its business operation. In order to accomplish this task, PT XYZ has realized that it needs to set up its leadership skills guidelines so that every leader poses the identical leadership style when handling its business Improving the Average Session …… (Jonny) 127  operation. In doing so, among several theories, the company has picked the power of leadership in which there are five levels of leaderships such as highly capable individual, contributing team member, competent manager, effective leader, and level 5 executive (Maxwell, 2013). This theory has been implemented in its leadership program pipelines as basic management program, supervisory program, the manager program, the senior manager program, the general manager program, and the executive program. Among these programs, they share the same leadership competency as their basis in developing people’s leadership skill naming leadership competency. According to Kaplan (2004), leadership competency model focuses on the specific competencies desired from leaders. In developing this model, among several theories, the company has selected Belker et al., (2012) in order to develop this competency. It is consisting of Vision and Business Sense (VBS), Competency Focus (CFO), Interpersonal Skill (ISK), Analysis and Judgment (AJU), Planning and Driving Action (PDA), Leading and Motivating (LMO), Teamwork (TWO), Drive, Courage, and Integrity (DCI). This competency has served as standard which based on Harris (2014), it can be the training design for ABC division to provide the suitable sessions to develop the needed leadership skill which are consisting of Effective First Line Manager (EFLM), Leading Empowered Team (LET), Situational Leadership (SL), Coaching and Counseling (C&C), Managing Resistance (MR), People Problem Solving (PPS), Execution Management (EM), Problem Solving and Decision Making (PROSDEM), Leading Effective Meeting (LEM), and Managing Up (MU). These sessionsare in line with what Belker et al., (2012) has said on the leadership skills that should be posed by a leader. Those sessions are further developed by ABC division using contemporary practices related to leadership practices in business operation which are further validated using pre-test to its leaders on whether it is suitable to be further delivered to the participants. After having its validation and approval from its top management, ABC division is about to conduct the training. However, previous research and literature related to this subject matter are still rare except Harris (2014), which has stressed out that there is the strong relationship between trainer style and learner orientation in predicting training outcomes. The training outcomes such as training satisfaction and outcomes, the company has decided to use internal and external facilitator among its group to deliver the subject matter in the class training to in order to ensure the training satisfaction and transfer. It has further set out 4.10 as the target to indicate that the training outcome has been delivered well. This index is further set out because based on Kirkpatrick (2009), this is the very first gateway for the participant in order to see whether the program is already suitable for them or not. This moment of truth is very important because it will determine whether the participant will pursue the other step of training effectiveness such as learning, behavior, and results. Furthermore, the company has long posed improvement culture where its people believe that anything in the company should and could be improved in order to achieve the higher level of achievement and satisfaction. There are several methods that the company has utilized especially in this paper, the company has decided to use PDCA cycle as proposed by McCarty (2005). Based on this method, there are four phases in order to improve the outcome of training effectiveness such as Plan, Do, Check, and Action. In the phase of Plan, the team will identify and analyze the problem which can be done by gathering data, analyze the gap, and develop the plan. After that, in this phase of Do is intended to implement the improvement plan in order to eliminate the root cause of the gap found in the previous phase. Next is Check that the team will compare the result with the target. If the result is satisfactory, then it can proceed to the next phase. Otherwise, they should return to the first phase. And the last is Action. This phase is for the team to standardize what solutions are effective to eliminate the root causes. By using this method, the company wishes to improve its training practices, so that it can also add new knowledge related to training management. It is especially the knowledge related to in-class training matters that is not only on what and how to 128   Journal The WINNERS, Vol. 17 No. 2, September 2016: 125-132  conduct the training but also importantly as proposed by Harris (2014) that the relationship between the trainer style and learner orientation on training outcome. METHODS In order to handle the problem, the team has adopted PDCA (Plan, Do, Check, and Action) cycle based on McCarty (2005). As described in the previous section, this cycle consists of four phases such as Plan, Do, Check, and Action and is widely used to improve any process in the business in order to gain higher achievement or better QCD (Quality, Cost, and Delivery) of its products or services. Due to its similarity to this method, it is also suitable to be used in order to improve the average session evaluation of the supervisory program of this research intended to be accomplished. The following section elaborates more on what to do during each step based on McCarty (2005). In the phase of Plan, the planning is done by conducting several steps such as (1) Gathering quantitative and qualitative data by using session and program evaluation format to see current condition. In order to accomplish this step, the team has distributed questionnaires to all participants of supervisory program batch 46 which consists of 30 persons from all PT XYZ’s affiliated companies. The questionnaires consist of three parts to evaluate Program Quality Index (PQI), Subject Content Index (SQI), and Facilitator Evaluation Index (FEI) of the program. Several exemplary questions of these questionnaires are according to Jonny (2016), such as (a) the program content meet its stated objectives, (b) the facilitators are able to maintain and encourage participation, (c) there is appropriate use of case discussion, examples, group work, and other means to aid understanding of the content, (d) the staffs are courteous, helpful and provided the trainees with professionalized attention, (e) the facilitators use practical illustration and or demonstrated an understanding of practical issues, (f) overall program is very helpful for the trainees in creating practical ideas to improve his or her performance. (2) Using qualitative data to find its roots causes by using cause and effect diagram. (3) Design the scenario improvement plan as the countermeasure on the problem. After having the improvement scenario, the team starts to implement the countermeasure to eliminate the root cause of the problem. In the phase of Do, all alternative solutions are generated and selected to find the best possible solution to be conducted. Furthermore, if in the field the selected solution is found to be not applicable, then the team will conduct small PDCA cycle as its problem identification and corrective action in order to make sure the plan is fully implemented. Next, the Check phase, the team is about to evaluate the result of all implemented solution on whether it is already effective or not. This phase is needed in order to ensure that all implemented solutions are really correlated as countermeasure so that the targeted training outcome from the company can be reached. If the case is not as it is, then the team may look back on the problem as described in phase plan. The Action phase is the final phase of the PDCA cycle. In this phase, all the effective solutions which have been proven in its efforts to achieve the targeted level are all to be standardized as the effective scenario. Therefore, it can be served as additional new knowledge as the contribution to the greater knowledge on training management especially in conducting the session in class. Although this method is familiar with manufacturing fields, however through many types of research and practical applications, it has been proved that it is also suitable in the service field. Therefore, it can also be served as an appropriate method for improving the training program. The company also finds this method is simple than other available methods. Therefore after having trained, the team starts using this method to improve the program. Improving the Average Session …… (Jonny) 129  RESULTS AND DISCUSSIONS In this section, the team elaborates more on the result of its effort to improve the supervisory program by using the PDCA cycle as their method. They have used training documentation of batch 46 as the base in using PDCA cycle. This batch consists of 30 people who are both already supervisors or supervisors to be that have been sent by their related companies to attend the supervisory program consisting of ten sessions and five-days training in the head office of PT XYZ. Those 30 people are then to be the respondent to the team in order to investigate and improve the program. The eligibility is based on their participation in the program as assigned by their companies. For additional information, some of them have also attended basic management program to describe their maturity on the management matters. This company has further renamed the program as the first-line manager to adjust their roles are first-line management role in the company. Furthermore, about the result, it has been further elaborated in the following session PDCA. In the Plan phase, the team has gathered qualitative and quantitative data from the two-form questionnaires that being feedback by the participants every time they finish the session and once the program has been completed. The first form of the questionnaire is named as session evaluation form in order to assess SCI and FEI and program evaluation to assess PQI. For assessing SCI and FEI, the team uses five-Likert measurement to let the participants give feedback from 1 (strongly disagree) to 5 (strongly agree). The questions used in this form are such as (1) the session content meets the stated objective, (2) the concept or skill presented will be helpful to the job, (3) there is appropriated use of case discussion, like group work to aid understanding of the subject, (4) the handout and or materials are designed and organized to aid understanding as SCI session. Then, in FEI session, the questions are (1) facilitator is well prepared and organized for the session, (2) facilitator demonstrates the competence in the subject matter, (3) Facilitator is able to communicate issues and concept clearly, (4) facilitator uses practical illustration and/or demonstrated an understanding of practical issues, (5) facilitators is able to maintain and encourage participation. For assessing PQI, another form using five-Likert measurement is also used in order to assess the effectiveness of the program which has already conducted. In that form, there are several questions such as (1) the program content meets the stated objectives, (2) the program is well composed and all arranged to support the effective learning process, (3) overall the program is very helpful in creating practical ideas to improve the performance as program effectiveness section. For management program, the questions are (1) staffs are courteous, (2) staffs are helpful, and (3) staff provides you with professionalized attention. While for facility related, there is one question which is the room set up is ready, and the equipment is in working order. From these forms, the team has gathered the questionnaire forms, input the feedback quantitatively into the system, and has calculated the result is giving. The following results such as PQI is 4,19, SCI is 3,95, and FEI is 3,93. From these results, it can be concluded that the program is not effective if the targeted index is 4,10 out of 5,00. Therefore, the team proceeds to the second step which is gathering the qualitative data that may be found in those forms left by the participants as input for the team. For the qualitative data gathered from those forms, there are many inputs on how the program should be conducted. This can be understood by reviewing their profile which can be classified as trainees with high learning orientation as introduced by Harris (2014). From those scattered qualitative feedbacks gathered from participants, the team utilizes the fishbone diagram or cause-effect diagram in order to analyze why the program has considered not effective yet by its participants as shown in Figure 1. 130   Journal The WINNERS, Vol. 17 No. 2, September 2016: 125-132  Figure 1 Qualitative Data which is Analyzed by Using Fishbone Diagram Figure 1 has revealed out that there are several root causes on why the program is not considered effective yet. First, the contents of the program are still considered as confusing because there is no framework for participants to figure it out on what is the program intended for them. Second, regarding the company policy to combine external and internal facilitators among the group, whoever is appointed as the facilitator, they should not be considered as new one although in fact, they are totally new. This is in line with the finding of Harris (2014) that the trainer competency is also matters regarding learner with high learning goal orientation. Third, the teaching strategies that used during delivering the subject content are considered boring which might make participants lose their focus on the matters. After having root causes validated, then the team also gathersanother qualitative data to explore more on how to improve the teaching method by conducting improvement scenario plan as described in Table 1. Table 1 Improvement Scenario Plan Voice of participants Mode of teaching method Slide Case Games Multimedia Slide contains typo and confusion X Need exercise to figure it out X Need some refreshments X Need audio visual to figure out the concept X From Table 1, it can be concluded that there should be several countermeasures to be related to slide, case, games, and multimedia in order to improve the teaching quality during the sessions which will be elaborated more in the following section. After the Plan phase, next phase is Do. After conducting the analysis, the team has come up with several improvements based on the improvement scenario plan such as (1) improving the teaching policy where facilitator arrangement should be a combination of 80% experienced and only 20% new with prior briefing to make them more confident and not seem really new about the session, (2) content should be based on framework that can be seen in Figure 2. Improving the Average Session …… (Jonny) 131  Figure 2 Framework of Supervisory Program Figure 2 gives the whole picture of the supervisory program. It is said that the program is about to improve the participants' leadership skill related on managing self, people, and the job with intended leadership competency and related section. By using this framework, the participants have a clearer understanding of the program. The content is also equipped with several teaching aids such as proper slide, case, games, and multimedia as planned in the scenario. After implementation on the next batch which is 47, the team pursues the next phase. After the implementation, in the phase of Check, the participants of the batch 47are asked to fill out feedback forms to the team which revealed out PQI is 4,55, SCI is 4,21 and FEI is 4,21. Based on these results, the program can be considered as effective and therefore it can proceed into the next phase. In this phase, Action, the team has standardized the several rules. First, facilitators are a combination of 80% experienced and only 20% new ones. This policy is needed in order to maintain the quality of knowledge transfer during the program. If the program is conducted by all new facilitators, then the quality of the program will be difficult to be managed. While if all the facilitators are experienced than the institution, it will not have any opportunity to do facilitator’s generation in case that those experienced ones decided to leave the institution for the new career. It cannot be managed by 50-50 rules because from the survey, the participants have stated that they need experienced facilitators over new ones since due to they are assigned to learn leadership skills, therefore, they need to get it from experience ones. However, if there is a new one that should be asked to deliver the material, the institution has made sure his/her readiness through rigorous briefing so that the quality of the program is well assured. Second, the content should be based on the framework. The survey also reveals that the content that has no adequate framework that will cause the participants find the difficulty in understanding the material. This phenomenon is understandable because the participants come from the company with holistic view culture. They are already trained to see the big picture through the framework of ideas. This is needed in order to make them buy in with the training. After understanding the framework, then they will believe in the content and make it easier for the facilitator to deliver the material. Third, several teaching aids that used are the slide, case, games, and multimedia. Some content of the leadership program are difficult to be understood, therefore, they need some aids to stimulate their understanding. Thanks for the sophisticated technology advancement because the facilitators can make the means to be used during delivering the material in the class such as slide, case, games, and multimedia. The use of these aids has proved the incremental of the session evaluation score, therefore, those aids are standardized as mandatory when the facilitator is about to deliver the material in the class. 132   Journal The WINNERS, Vol. 17 No. 2, September 2016: 125-132  CONCLUSIONS After conducting the research, the team has found that the reasons why the program is below target 4,10 score are related to the new facilitator, no framework, and teaching aids. Therefore, after doing these improvements, the score is improving to above targeted 4,10 score. The additional contribution to the knowledge that can be derived from this paper are (1) trainer combination should be paid attention, (2) use of framework, and (3) use of teaching aids such as proper slide, case, games, and multimedia. However, this research has some limitation. First, the participants as research objects are different from batch 46 and 47 which may result in biased results. Second, the questionnaires forms should be further validated as input for future research. The last, the interaction of those solutions has been neglected as its practicality reason which in the future research it can be input for better research. REFERENCES Belker, L. B., McCormick, J., & Topchik, G. S. (2012). The First-Time Manager. US: AMACOM. Harris, T. B, Wonjoon, C., Hutchins, H. M., & Chiaburu, D. S. (2014). Do Trainer Style and Learner Orientation Predict Training Outcomes? Journal of Workplace Learning, 26(5), 331-344. Hessen, S. C. (2015). ISO 9001:2015 and the 8 Quality Management Principles to take you to the Head of the Class. Retrieved May 31st 2016 from http://pilgrimquality.com/blog/iso90012015-quality-mgmt-principles/ Jonny. (2016). Training Effectiveness Analysis at PT XYZ using Kirkpatric Model and Return of Training Investment (ROTI). Journal of Binus Business Review, 7(2), 137-141. Kaplan, R. S., & Norton, D. P. (2004). Strategy Maps: Converting Intangible Assets into Tangible Outcomes. Massachusetts, USA: Havard Business School Publishing Corporation. Kirkpatrick, J., & Kirkpatrick, W. K. (2009). The Kirkpatrick Four Levels: A Fresh Look After 50 Years 1959-2009. Retrieved February 11th 2016 from http://www.kirkpatrickpatners.com/portals/0/resources/kirkpatrick%20Four%20Levels%20wh ite%20paper.pdf. Maxwell, J. C. (2013). The 5 Levels of Leadership. US: Hodder & Stoughton General Division. McCarty, T., Daniels, L., Bremer, M., & Gupta, P. (2005). The Six Sigma Black Belt Handbook. USA: McGraw Hill. Sutyanto, D. (2013). Indonesia’s Top Ten Companies Ranked by Largest Market Capitalization. Retrieved May 31st 2015 from http://www.indonesia-investments.com/finance/financial- columns/indonesias-top-ten-companies-ranked-by-largest-market-capitalization/item637. The World Bank. (2016, 31 May). Market Capitalization of Listed Domestic Companies (% GDP). Retrived May 31st May 2016 from http://data.worldbank.org/indicator/CM.MKT.LCAP.CD