AN INSTRUMENT FOR THE ASSESSMENT OF LEARNING DIMENSIONS: A PROGRESS REPORT ON THE LEARNING DIMENSION SCALE (LDS)1 Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 122 AN INSTRUMENT FOR THE ASSESSMENT OF LEARNING DIMENSIONS: A PROGRESS REPORT ON THE LEARNING DIMENSION SCALE (LDS)1 Steven W. Lamb, Indiana State University Samuel C. Certo, Indiana State University ABSTRACT 1 A version of this paper has been developed into a grant proposal, the final disposition of which is still pending. INTRODUCTION The use of experiential materials by instructors in various learning situations has established itself as more than a passing instructional fad. In the area of management, as with many other business areas, although relatively few experientially oriented texts were published in the more distant past (19), an in-creasing number has appeared in the more immediate (1, 9, 14) and very recent pasts (5, 8, 22). During this same time frame, the focus of experientially related research and other scholarly works has seemed to shift somewhat from “should an instructor use various experientially oriented versus non-experientially oriented pedagogic devices (2, 4)”, to “how should one best use and/or design experiential materials (13, 23)”. In line with this trend, the purpose of this paper is threefold: 1) to summarize a movement over the past few years to access and use individual learning dimensions in the design and conduct of experiential learning materials; 2) to present an intended step forward by introducing a proposal for the development of the Learning Dimension Scale (LDS), an instrument which will better enable instructors to evaluate and use individual learning dimensions to enhance the effectiveness and efficiency of experiential learning materials, and; 3) to encourage others to pursue similar instruments due to the worthwhileness and high need for such measurement tools. THE ASSESSMENT OF LEARNING DIMENSIONS IN THE PAST A learning dimension is defined as a major characteristic of the process individuals use in order to acquire new knowledge. It is therefore, fundamental that in order to discuss these dimensions one must first understand how people acquire new knowledge (learn). According to a model synthesized by Kolb (16), the learning process is a four-stage cycle. Concrete experience (Stage One or CE) serves as the foundation for reflections and observations (Stage Two or RO) which result in abstract concepts (Stage Three or AC) which are then tested through active experimentation (Stage Four or AE) in new situations. These four stages are continually repeated as the cycle depicted in Figure 1 implies. The Learning Styles Inventory (LSI) is probably the most pervasive instrument which has been designed to measure the degree to which individuals use concrete experience, reflective observation, abstract concepts, and active experimentation in their learning processes (17). Basically, the LSI is purported to determine: 1) AC, CE, AE, and RO scores for individuals dependent upon the relative emphasis they place on each stage during the learning process, and; 2) scores reflecting the relative emphasis individuals place on abstractness versus concreteness (AC score minus CE score) and activeness versus reflectiveness (AE score minus RO score). The LSI and related scoring procedures are presented in Figure 2. Unfortunately, the LSI has not withstood the test of close empirical scrutiny. Freedman and Stumpf (10) administered the LSI to one group of 1179 students on a one time basis and to a second group of 101 students on a test-retest basis. These authors state that overall, the LSI is a worthwhile idea but that the credibility of the LSI is seriously suspect due to its rather unreliable nature. These researchers conclude both with the thought that LSI instrument bias is related to completion and scoring procedures and with the question of what, if anything, does the LSI measure?” More recently, similar conclusions have been reinforced and extended by Freedman and Stumpf (11). Certo and Lamb (6) compared LSI results generated by student response to LSI results generated by a random numbers table. This comparison led the authors to conclude that a significant degree of artificial correlation built into the LSI rendered the instrument deficient. A related study by Certo and Lamb (7) led to similar conclusions. Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 123 Evidence concerning the high value of a sound instrument such as the LSI is not difficult to gather. According to Hall, Bowen, Lewicki, and Hall, (12, p. 27) “understanding one’s learning style helps the manager to seek out learning and problem-solving situations which are likely to contribute most to his or her continued development." In addition, preferred learning styles can be studied against various teaching methodologies in order to enhance the learning situation (3). Lastly, as evidenced by its inclusion in two popular organizational behavior texts (12, 18), the LSI can serve as excellent basis for classroom discussion. THE ASSESSMENT OF LEARNING DIMENSIONS IN THE FUTURE: THE LDS Initial Item Identification and Development The first task in the development of the LDS will be to identify a set of questions or items which is capable of assessing an individual’s identification with the AC, CE, AE, RD learning dimensions. Items will be generated which have little mystic associated with them and clearly characterize one dimension or another. Newman (20) reports that within the development of his Perceived Work Environment (PWE) instrument, great care was taken to assure that the questionnaire items were as nonevaluative as possible by developing items that maximized description and minimized evaluation. The items will be developed using a seven-point Likert scale instead of using a forced rank instrument. The Likert scale will not force any negative correlation thus allowing any correlation that exists between dimensions to be estimated. Attention, however, will also be focused on the limitations of the Likert scale such as consistency of responses through the development of a mental set. Items will not be developed so that subjects are forced to choose between dimensions. A question phrased in such a manner could also develop an artificial mental set in a students’ mind that could create a bias in that individuals response to other items. It may be that analysis of the result does not support the bipolar state of learning that related instrumentation in this area espouses. It might be that learning is so interdependent that strong development in one dimension requires at least moderate development of all other dimensions. Item Validity In order to determine whether or not a potential Item is capable of assessing an individual’s identification with a learning dimension the following procedure will be used. First, the learning model will be presented to various sets of subjects and an attempt will be made to thoroughly acquaint them with the meaning and definition of each of the dimensions. Next, subjects will be asked to classify each of the potential questions under its appropriate category (Ac, CE, AE, or RO) and rate the ease with which this classification is made. Only those questions that a majority of subjects are capable of classifying correctly and are judged to be easily classified will be retained. Kerlinger (15) States that “content validation consists essentially Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 124 in judgment. Alone or with others, one judges the representativeness of the items.” If the subjects as a majority are able to determine that a question is measuring what it is intended to measure, a significant improvement over past instrumentation in this area will be achieved. Item Reliability After the set of questions has been identified, an analysis of the instrument will be judged using the following techniques. A test-retest procedure will be used to determine the consistency of the responses to the instrument. That is, the same questionnaire will be administered to the same set of individuals after an appropriate period of time has passed. Correlation matrices will be formed using the test-retest scores for each item. These reliability estimates will need to be fairly high, given Nunnallv’s (18) argument that test- retest reliabilities are frequently inflated. Again, if any item were to demonstrate a lack of consistency, it would need to be eliminated. However, the early attempt to select questions that demonstrated high content validity should help reduce the possibility of having items in the instrument with little reliability. A correlation matrix for the test-retest results will also be run for totals of each of the four dimensions. Construct Validity In order for the instrument to be judged valid, it will have to be four dimensional. That is not to say that the dimensions need be uncorrelated. However, the correlation of the items within the dimensions should be stronger than the correlation of the items among dimensions. The dimensionality of the instrument shall be determined using both correlation analysis and factor analysis techniques. In order to determine whether or not the questions effectively identify four dimensions (independent or otherwise) using correlation analysis, data will first be analyzed through the development of two matrices. The first matrix will report the average correlation of each variable with the other variables in each of the four hypothesized dimensions. Ideally each variable should have a strong positive average correlation with the items within its own construct as compared to the Items in the other three dimensions. For clarity, if there exists twenty-six questions within the questionnaire, the matrix would be of size four by twenty-six. If an item has weak average correlation with other items within its own dimension or is more strongly correlated with items within other dimensions that item would need to be reclassified or eliminated. The second matrix will report the average correlation value of all the paired variables within each of the four hypothesized dimensions (those values will appear on the diagonal in this matrix) and the average correlation of all paired variables between each of the paired dimensions (these values will appear as the off- diagonal items). The four correlation values on the diagonal of the matrix will be found by averaging the correlations of the variables within each of the hypothesized dimensions, whereas the six correlation values on the off-diagonal will be found by averaging the correlations of the variables between each pair of dimensions. Ideally, the table will be able to demonstrate that the average level of interaction between variables within each respective dimension is much higher than the average correlation of variables among dimensions. That is, the correlations on the diagonals would all hopefully be higher than any single correlation on the off-diagonal. This matrix would be of size four by four. To supplement the findings of the correlation analysis, factor analysis techniques will also be used. Specifically, principal factoring using principal component analysis and Varimax four factor rotation procedures will be used. Ideally, each factor created will successfully identify one of the hypothesized dimensions. The items comprising each of the hypothesized dimensions should load heavily on its dimension while every other item within the instrument should have a small correlation value with that factor. Items that load on several dimensions or on none will have to be eliminated. CONCLUSIONS This paper has presented a progress report on the development of an instrument called the LDS. The development of this instrument has been planned according to generally accepted validation and reliability procedures. In addition to resulting in a finalized experiential learning measurement tool, it is hoped that this article will encourage the development of similar instruments by others. REFERENCES (1) Beatty, Richard W. and Craig Eric Schreier, Personnel Administration: An Experiential Skill-Building Approach (Reading: Addison- Wesley, 1977). (2) Belasco, J.A., A.M. Glassman and J.A. Alutto, “Experiential Learning: Some Classroom Results’ Academy of Management Proceedings 1973, pp. 235- 246. (3) Brenenstuhl, Daniel C. and Ralph F. Catalanello, ‘The Impact of Three Pedagogue Techniques on Learning” Journal of Experiential Learning and Simulation, Vol. 1, No. 3, 1979, pp. 211-225. (4) Butler, J.L. and B.L. Keys, ‘A Comparative Study of Simulation and Traditional Methods of Supervisory Training in Human Resource Development” Academy of Management Proceedings 1973, pp. 302-305. (5) Certo, Samuel C. and Lee A. Graf, Experiencing Modern Management: A Workbook of Study Activities (Dubuque: William C. Brown, 1980). (6) Certo, Samuel C. and Steven W. Lamb, "Identification and Measurement of Instrument Bias within the Learning Styles Inventory Through a Monte Carlo Technique” in Ray, Dennis F. and Thad B. Green (eds.) Proceedings of Southern Management Association, 1979, pp. 22-24. (7) Certo, Samuel C. and Steven W. Lamb “An Investigation of Bias Within the Learning Styles Inventory Through Factor Analysis’ Journal of Experiential Learning and Simulation, Vol. 2, No. 1, 1980, pp. 1-7. (8) Doktor, Robert, Alfred Edge, and Lave Kelley, Experiencing Management: Active Learning for Large Classes (Honolulu: Management Publishing, Inc. , 1980) (9) Finch, Frederic E., Halsey R. Jones, and Joseph A. Litterer, Managing for Organizational Effectiveness: An Experiential Approach (New York: McGraw-Hill, 1976). Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 125 (10) Freedman, Richard D. and Stephen A. Stumpf ‘What Can One Learn from the Learning Styles Inventory?” Academy of Management Journal Vol. 21, No. 2, 1978, pp. 275-282. (11) Freedman, Richard D. and Stephen A. Stumpf “Learning Style Theory: Less than Meets the Eye” Academy of Management Review, Vol. 5, No. 3, 1980, pp. 445-447. (12) Hall, D., D. Bowen, R. Lewicki, and F. Hall, Experiences in Management and Organizational Behavior (Chicago: St. Clair Press, 1976). (13) Jackson, John H. , William J. Sawaya, and Douglass K. Hawes “Using an Experimental Approach to Experiential Learning: A Leadership Example Journal of Experiential Learning and Simulation, Vol. 1, No. 4, 1979, pp. 257-265. (14) Kast, Fremont E. and James E. Rosenzweig, Experiential Exercises and Cases in Management (New York: McGraw-Hill, 1976). (15) Kerlinger, Fred N., Foundations of Behavioral Research, New York: Holt-Rinehart, Winston) p. 458. (16) Kolb, David A., “On Management and the Learning Process’ in Kolb, David A., Irwin M. Rubin, and James M. McIntyre (EDS) Organizational Psychology: A Book of Readings, (Englewood Clipp: Prentice Hall, 1974, 2nd Ed. pp. 27-41). (17) Kolb, David A., Individual Learning Styles and the Learning Process. Working paper No. 535-71, Sloan School of Management, MIT, 1971. (18) Kolb, David A., Irwin M. Rubin, and James M. McIntyre, Organizational Psychology: An Experiential Approach, (Englewood Clipp: Prentice- Hall, 3rd Ed., 1979). (19) Maier, Norman R.F., Allen R. Solem, and Ayesha A. Maier, Supervisory and Executive Development: A Manual for Role Playing, (New York: John Wiley and Sons, Inc., 1957). (20) Newman, John E. ‘Development of a Measurement of Perceived Work Environment (PWE)” Academy of Management Journal, Vol. 20, No. 4, 1977, pp. 520- 534. (21) Nunnally, J.C., Psychomethic Theory, (New York: McGraw-Hill, 1967). (22) Veiga, John F. and John N. Yanouzas, The Dynamics of Organization Theory: Gaining a Macro Perspective, (St. Paul: West, 1979). (23) Warrick, D.D., Phillip L. Hunsaker, Curtis W. Cook, and Steve Altman “Debriefing Experiential Learning Exercises” Journal of Experiential Learning and Simulation, Vol. 1, No. 2, 1979, pp. 91-100. Table of Contents Volume 8, 1981 The Promotion: Human Sexuality in Organization The Simulation of Chaos Leadership Development in a Simulated Urban/Suburban (U/S) Environment Tomed: A Computer Game Emphasizing Social Responsibility/or/why the Pop-Top Can? Integrated Brain Activity and the Manager's Job: Utilizing the Troika Model What does R2 Have to do with a Product Management Course? An Analysis of the Effects of Jungian Problem-Solving Style Dimensions on Marketing Decisions Bargaining Behavior in Personal Selling and Buying Exchanges Extending the Simulation Product Life Cycle A Generalized Algorithm for Designing and Developing Business Simulations Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises Pygmalion and Perception: An Experiential Exercise Behavioral Consequences of Reward Regarding Employee Absenteeism in an Industrial Setting: An Operant Conditioning Approach The Simlab Program: The Use of Experimental Simulation and Process Analysis for the Development of Management and Organizations Designs for Research on Simulation-Games, Cases, and Other Experiential Exercises The Role of Students in The Case Method Weaknesses in Research Design Critical Variables in Research on the Educational Value of Management Games Research Questions for Cases Research on the Learning Effectiveness of Business Simulation Games - A Review of the State of the Science The Effects of Valuation Techniques on Holding Cost During Inflationary Periods: A Simulation Exercise The Operations Simulation - A Study in Game Development Applying Guided Design to the Production/Operations Management Course: A Progress Report and Evaluation Terminal Data Entry and Retrieval Systems Simulations and Microprocessors Microcomputers and Related Technology for Simulation Gaming Microcomputers - A New Technology for Innovations in Business Simulations Microprocessor Controlled Interactive Video Simulation Business Game Design: From Theory to Practice The Success of a Computerized Simulation in Microeconomic Pedagogy The Test Preview Game: Applying the Game Show Format Providing a Real World View of the Personal Function: A Simulation Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared A Management Development Program Based on the Experiential Learning Model Participant Type Differences in Response to Experiential Methods: An Informal Look Preparing Student Groups to Participate in Experiential Group Projects: An Organizational Development Approach An Instrument for the Assessment of Learning Dimensions: A Progress Report of the Learning Dimension Scale (LDS) An Empirical Analysis Relating the Learning Style Inventory to Memory and Logical Ability Teaching Styles in Simulation Experiential Learning Versus Traditional Teaching Styles Student Perceptions of Effective Teaching Behaviors Problems in Evaluation of Experiential Learning in Management Education Students' Perceptions of Learning by Simulation A Relative Evaluation of Experiential and Simulation Learning in Terms of Perceptions of Effected Changes in Students Overview of Computer Based Business Games in Business Policy Classes Behavioral Decision Theory and Business Policy Giving Accounting Students Writing Experience as Job Preparation Getting to First Base with MBO: An exercise for Writing and Evaluating Objectives Dimensions of Conflict in Experiential Learning Consumer Alienation and Perceived Relevance of the Business Simulation Using the Self-Reference Criterion to Simulate Culture in Internationalized Business Course Experiential Learning in a Cross Cultural Setting- The Practice of Simulation Approach to Business Education in Japanese Universities Student' Perceptions of the Use of a Computerized Simulation in Teaching Management Information Systems Using the Case Study Approach to Develop a Microcomputer Bases, Fully Integrated, Data Base Driven, Management Information System The Case Study as a Tool for Organizational Change: Applying the Steel Ax to the Designers of Management Information Systems Toward a Theory of Teaching Business Policy A Case Study in the Use of Experiential Learning (A Management Game Simulation) to Enhance Student Understanding of Strategy Evaluation and Policy Formulation Suggestions for Integration of the Business Administration Core Publishing Opportunities and Requirements for Business Simulation and Experiential Learning Materials How do we Apply Experiential Learning Intercollegiate Case Competitions for M.B.A. Students: Initiation and Implementation Teaching Business Policy and Strategy Using the Incident Process The Learning Co-Op Approach to the Core Policy Course Meeting the Managerial Skill Shortage - Is Academia Up to It? An Empirical Analysis of Experiential Learning Reinforcement International Experiential Learning: Experience is the Best Teacher Encouraging Student Participation During International Academic Programs European Summer Study Program: Can you, Should you, What Does it Take? Travel Seminars in Europe: How can I Direct One? International Experiential Learning: Student Evaluation ABSEL: Empirical Findings on the State of the Association Sensitivity of Performance Scores in Business Simulations Improving the Learning of a Business Simulation Game by Increasing the Process Content Student Participation in Deciding Performance Criteria for Grading in an OB Course: An Exercise and a Case Study Markup for Profit: A Simulated Self-Administered Experience in Retail Pricing The Investment Decision Game: An Experiential Learning Approach to Stock Market Decisions Through Gaming CHIPS: A Marketing Channels management Game External Validation: An Experimental Approach to Determining the Worth of Simulation Games Teaching Performance Appraisal Skills: An Experiential Approach In Support of Experiential learning: Results of a Follow-Up Survey The Introductory Management Course: Taking Theory Application One Step Further Decision Efficiency and Effectiveness in a Business Simulation The Implications of Cognitive Processing Variables and the Complex Decision Simulating the Simulation for Enhanced Player Rationality Simulation/Experiential Learning Audit