MEASURING “QUALITY” IN MANAGEMENT OF BUSINESS PEDAGOGY Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 215 MEASURING “QUALITY” IN MANAGEMENT OF BUSINESS PEDAGOGY George D. Kesling and Gerald L. Cleveland School of Business and Economics, Central Washington University ABSTRACT The focus of this year’s theme of “Total Quality Management in Business Pedagogy" for academics, business and government promotes an attempt to measure “quality from these sources. A simplistic model is demonstrated using cases prepared for the Small Business Administration to stimulate thought and discussion on the evaluation of "quality”. INTRODUCTION "Quality" management of the pedagogy in the changing nature of business requires continued evaluation of the methods and individuals. The measurement of “quality” in the pedagogy is essential to controlling and improving the process. Students, faculty, school administrators, business executives and government officials all contribute to the evaluation in one way or another. Probably the most common method is to use some sort of numerical evaluation as information in the subjective evaluation that is ultimately made. The simplistic model presented in this paper asks participants to evaluate an undefined term, quality," of a project, course, individual, or curriculum on a scale of one to five with five being best and combines equally weighted objective and subjective evaluations from available sources. Comparing evaluations can help select competing alternatives. MODEL DEMONSTRATION The demonstration uses equally weighted evaluations from the students, instructor, business and government to evaluate the "quality” of five projects recommending computing equipment, prepared by small groups of university students and submitted to the Small Business Administration (SBA) for consideration for funding (see Table 1). Students’ input is based on their experience with different methods and instructors. Their average evaluation of the "value” of the course was used to assign the quality” ratings in this example. The instructor’s input is based on three subjective and two objective evaluations recorded as numerical scores. Evaluation of the project by the business is based on the arithmetic average of an evaluation form rating five aspects of the project. Input from the SBA consists of a single subjective evaluation based on the information it has received. Since the decision of which projects to fund rests with government officials, accuracy of the evaluation should be of particular importance to the "quality” of its management. The decision, of course, can depend on many factors not included in the numerical ratings. Although this demonstration evaluates projects, pedagogy can also be evaluated using similar inputs. DISCUSSION “Quality" in management of business pedagogy is a measure of management’s success in choosing the most valuable skills, knowledge, methods and individuals that constitute the pedagogy. The evaluation of teaching methods, for example, can play an important role in improving pedagogy. The value of a computer simulation game must sometimes be compared to that of an actual business experience as pedagogy is adapted to suit the changing business environment. Good measurements are essential in achieving accurate evaluations. William Zikmund (1991) lists three criteria for good measurement: reliability, validity and sensitivity. Reliability measures the extent to which measures yield consistent results. If the evaluation method produces the same result when it is repeated by the same evaluators for the same projects, it is internally consistent. If the evaluation method produces similar results when it is repeated for the same projects by other evaluators, it is externally consistent. The mathematical property of multiple measurements that provides the basis for the central limit theorem tends to improve consistency as the number of measurements increases. For internal consistency, this means breaking the evaluation into components usually improves consistency. It also allows the measurement o consistency. By the same principle, increasing the number of evaluators usually increases external consistency. Validity measures the extent to which a measurement measures what it is supposed to measure. This is particularly important, yet difficult, in measurements of business pedagogy. Value and quality, for example, have different meanings; quality refers to performance while value compares benefits to cost. In some situations, the value of a project, individual, method or pedagogy can be low even though the quality is high; even though the performance was excellent, it was not the most valuable thing to do with the resources. Value is usually computed by dividing the quality of the performance by its cost. Quality management of business pedagogy requires selection of the most valuable methods. Sensitivity measures an instrument’s (such as an evaluation form’s) ability to distinguish small changes. Reluctance of faculty members to make distinctions between the performance of colleagues, for example, reduces sensitivity. Increasing the range of possible scores on an evaluation form tends to improve sensitivity. The model used to combine measurements can also have a significant impact on the evaluation. Ranking by the average of several factors, for example, may produce different results than setting minimal requirements for these factors. This is demonstrated in the ratings in Table I; ranking by averages leaves project 2 with the lowest evaluation, while setting a minimal standard of two in each category eliminates project 3, yet finds project 2 acceptable. The model used by the American Assembly of Collegiate Schools of Business combines these methods by setting minimal standards in several areas, while combining several factors from several sources for an overall rating. CONCLUSION Although many good evaluation forms and methods are used to help evaluate the "quality of business pedagogy, the final judgement is usually at least partially subjective and can vary widely between individuals. Consequently, accuracy in the method is a problem. There are a few principles that can help improve accuracy: Numerical models often do not appropriately weight all factors. Increasing the number of individual sources tends to improve reliability. Increasing the number of factors in the measurement from each source also tends to improve reliability. Identifying exactly what is being measured helps improve validity. Increasing the range of possible scores tends to improve sensitivity. These principles, of course, must be limited in practical applications. The model and criteria used in the evaluation can have a significant impact on the results. The example using SBA cases demonstrates a simplistic way of combining evaluations in a numerical model and illustrates the impact of different evaluation criteria. Although numerical measurements and evaluation models do differ and none is completely accurate, most evidence implies that the development of such methods is one of the most promising ways of improving the "quality" in management of business pedagogy. REFERENCES Zikmund, W. G. (1991) Business Research Methods, (3rd ed.) Orlando, FL: The Dryden Press. Table of Contents Volume 19, 1992 The Pedagogical Utility of a Management Simulation Game in a Business Policy Course Modeling Economic Environments in Business Simulations: Some Comparisons and Recommendations Experiential Learning and TQM Principles: Teaching Behavioral Science in a Business School The Use of Poster Presentations as a Final Project in the Business Policy Course The Device Business: A Management Simulation for Strategy Formulation The Advantages of Experiential learning in the Auditing curriculum A Framework for the Identification of Moderated and Mediated Performance consequences of Pedagogical Alternatives Simulating Qualitative Research Relating to Values and Lifestyle Segmentation A New Market Demand Model for Business Simulators Interactive Optimization Using the Method of Relative Improvement Preferences: Methodology and Empirical Evaluation The Service Trainer Simulation Benefits of Internet Computer Networks for ABSEL Members How Should We Measure Experiential Learning? An Assessment of Simulation Usage in Management Accounting Courses The Influence of Myers-Briggs Type and Group Dynamics on Simulation Performance Effective Leadership Behavior in the Desert Storm Arena: An Application of the Vroom-Yetton-Jago Model Installing and Consolidating Work-Team Values: The Effects of a Multicultural Outdoors Experiential Program Key Determinants and Decision performance in a Business Simulations and Experiential Learning environment Multi-Cultural Experiential Learning: A Computer Simulation in Indonesia Attitudes Toward and Emotions Related to Women as Managers: A Replication and Beyond Scenario Approach to Simulating Consumer Expenditures: A Cross-Cultural Analysis Insights into Ethical Decision making Activities and Organizational Performance: A Management Simulation Analysis of College Students and Managers Evaluating a Business Simulation Program for Joint Venture Negotiation and Management Teaching Business Interviewing Strategies with an Experiential Approach Cooperative Learning Across the Business Curriculum Modeling Total Quality Elements into a Strategy-Oriented Simulation Teaching Business Decision making using a Simulator Extending the Educational Utility of a Simulated Competition within the Confines of an Established Undergraduate Marketing Curriculum Expert Systems Versus Traditional Methods for Teaching Accounting Issues Confidence Extremes Diminish Quality Performance in a Total Enterprise Simulation Can Ethics Be Taught? A Simulation Tests a Traditional Ethics Pedagogy Through The Looking Glass, Inc.: Organizational Climate Research as Experiential Pedagogy The use of Cluster Analysis for Business Game Performance Analysis Power and Ethnicity: An Experiential Learning Exercise (How to Sensitize Students to Diversity) Directed Development of Critical and Creative Thinking Skills for Case Analysis Implementing Total Quality Management in a Computerized Business Simulation Product Quality in Business Simulations Use of Simulation for Ethics Education in Management Satisfying the University's Customers through Total Quality Management Instruction: A Case Study Personality Characteristics and Group Performance in Total Enterprise Simulations Concepts of Interval Estimation and Quality Control Charts via Computer Simulated Sampling An Examination of the Effect of Team Cohesion, Player Attitude, and Performance Expectations on Simulation Performance Results The Effectiveness of Inventory Management and Production Scheduling Training in a Total Quality Management Environment Peer Group Indicators of the External Validity of Business Games: A five-year Longitudinal Study Political Strategies and Personal Actions Does Practice make Perfect? Observations on Simulation Trial The Use of A Non-Business Computer Simulation to Teach Marketing Management BankPro Commercial Bank Simulation The Production Game Pursuing Excellence: Work Strengths, Assets and relevant Values (An Exercise) Test of a Short Outward Bound Experience for College Students What is it that we want Student To Learn? Using Two TQM Philosophies when Playing Blackjack The Lagged Effects of Decision Variables on Financial Performance Measures Used in Two business Simulations The Development of an Experiential Exercise for Career Planning and Effective Job search Performance Measuring Quality in Management of Business Pedagogy Exploring Quality and Productivity Improvement: Using and Experiential Process Toward a Generalized Architecture for Intelligent Reactive Management Systems The Quality Game TQX: Using Expert Systems to Improve Training in Total Quality Management A Simulation of the Effect of the Medicaid Payment Lag on the Financial Position of Community Pharmacies The Use of a Board of Directors to Evaluate and Validate Decisions in a Competitive Graduate Management Simulation Course The Impact of Academic Dishonesty on Business Simulations and Experiential Learning Activities Picture Project: An Experience of Icebreaking and/or Decision-Making An Effective Role-playing Exercise for Teaching Requisite TQM Supervisory Attitudes/Behavior Computerized Management Simulations and Some Correlates of Students' Satisfaction The Timing and Stability of Reactions to Market Structure in a Single Player Simulation Environment A Graphics Application Extension for a Simulated Decision Support System Environment Aspects of a Group Project Utilizing Actual Business data and a Computerized Accounting System The Role of Universities' Extended Learning Department in Assisting Organizations Implement Total Quality Management A three-dimensional Learning Experience to Develop Total Quality Management Skills Estimating Quality Costs by Computer Simulation Applications and Examples of Quality Control Software Assessing Business Pedagogy A Demonstration of an Experiential Process for Exploring Quality and Productivity Improvement Processes An Action-Ethics Dilemma: A Demonstration Org Sim Jr.: A 2-3 Hour Version of the 2-Day Blanchard/Murrell Organization Simulation Building Buildings: An Experiential Exercise in Organizational Structure, Communication, Leadership and Group Dynamics Expert Systems for Organization Design: A Demonstration A Demonstration of Product Quality in a Business Simulation: Version 2 of CEO Two Revolutions, Total Quality Leadership, and the Baldy: The Story of Milliken Total Quality after the Award - The Xerox Story Building a Competitive Advantage through Customer Satisfaction and Reengineering Human Resource Planning: Managing the Only Renewable Resource for a Competitive Advantage Using Simulations to Teach International Issues: An analysis of the Multinational Management Game's Learning Environment Quality Function Deployment: A Tutorial Care and Nurturing of Teams Making a Good Thing Better: Adding TQM to Participative management Empowering Organizations to Redesign and Transform Themselves - Eastman Chemical Using Simulations in Field Management Development of an International Life Insurance Company Reinforcing the TQ Environment via Simulation Transforming a Business College into a Total Quality College