THE RESEARCH/TEACHING INTERFACE: TURNING A PRETEST INTO AN EXPERIENTIAL EXERCISE Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 235 THE RESEARCH/TEACHING INTERFACE: TURNING A PRETEST INTO AN EXPERIENTIAL EXERCISE Patriya S. Tansuhaj, Washington State University Roxanne Stell, Oklahoma State University ABSTRACT The benefits of student pretests as an experiential learning exercise are discussed with an illustration. Positive results were noted from implementation of the suggested method. INTRODUCTION Many researchers turn to students to pretest their questionnaire instruments and even to obtain actual data for their studies. Evaluations of students as surrogates in experimental and survey research were discussed extensively in Cunningham, Anderson and Murphy [1]; Dunne, Lund and Luchsinger [3); and Lamb and Stem [5). Those articles focused mainly on the validity of research results. There are yet ethical issues related to using students. Such as, what can be done to make students also benefit (learn from the research? And, how should the students’ rights related to participating in the instructor’s research be protected? Tybout and Zaltman [6] stated that subjects have the same rights as those of the consumer: right to choose, right to safety, right to be informed, and right to redress. In many cases, if the instructor is sensitive about these issues and plans ahead, he or she can build his (her) study into the course schedule, and thus minimize ethical problems. This paper discusses how a questionnaire pretest can be a good learning instrument for students while providing the instructor with high quality feedback from those students. The research/teaching interface method suggested here was used successfully at a large mid-western university. The procedures for implementation, assessment, and conditions under which the method would be most useful are discussed. BENEFITS OF USING STUDENTS FOR PRETESTING While pretesting has been highly recommended as a part of questionnaire design, it is often under-utilized or conducted haphazardly by researchers. A pretest is often conducted with subjects not representative of the actual sample (i.e. student or housewives). However, student pretesting can offer several advantages for both research and instructional objectives. Using students is more convenient and less expensive than using a subsample of the population. In addition, it is usually difficult to obtain a sample of respondents from certain populations (i.e. business executives). In situations where the population is limited, the researcher may not wish to chance losing valuable data by pretesting the instrument with his or her limited sample. Dillman [2] has suggested using colleagues, subsamples of the actual population, or potential “users” of the data for pretesting measurement instruments. Students may be treated as potential “users” if they are provided with adequate knowledge of the relevant research topics and are prepared to play the role of “users” such as business executives or administrators. Through the experiential exercise, students are able to become acquainted with such knowledge and information. They may also act as surrogates for individuals who might otherwise be considered potential “users” of the data or research results. Not only will the pretest yield high quality results, but it can also become an effective learning instrument in the classroom. An approach to maximize benefits of student pretesting, for both the researcher as well as to the student is discussed in the next section. TURNING A PRETEST INTO EXPERIENTIAL LEARNING A common practice in getting students to participate in a researcher’s study is to pass out questionnaires in a classroom with little effort devoted to discussion of the topic or content of the study. This practice is often necessary for research that requires avoiding potential external biases (i.e. experimental research). Unfortunately, in studies where these external biases are not very critical, researchers and/or instructors often fail to consider how much their students can and should learn from the study. Instead of simply handing out the researcher’s questionnaire instruments for students to complete in a hurry, a better alternative is suggested. The method includes three functions: 1) prior discussion or lecture on the topic related to the pretest, 2) a scenario or case development to lead students into a familiar situation and task, and 3) filling out the questionnaire (in class or outside) while playing a role stated in the scenario. According to Sewall [5], protection of human subjects becomes important whenever experiential learning for research or classroom instruction is used due to potential social and psychological effects. This approach attempts to minimize ethical problems. Not only do the students gain knowledge, but there is less stress than what might occur if students had to respond to questions that they could not answer. To protect their right to choose, we suggest making it a voluntary assignment for extra credit. Their familiarity with the topic, involvement through role-playing, and the opportunity to receive extra Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 236 credit should be significant motivating factors for most students to participate and to do a good job on a voluntary basis. Furthermore, it would be ideal if the instructor and/or researcher plans his (her) research timetable in such a manner that the findings are ready to be shared, at least briefly or tentatively, with the students by the end of the course. To describe the detailed activities and tasks involved in the approach, the following section reports how the method was implemented in the spring semester of 1984. AN ILLUSTRATION To clarify our previous discussion, an example is provided in this section. A pretest was conducted with students taking International Marketing in spring 1984 at a large mid- western state university. One of the authors had drafts of two sets of questionnaires to survey firms regarding their use of foreign trade zoned in their international logistics systems. Student pretesting was extremely useful in the present situation for a variety of reasons. First, there exist only a limited number of foreign trade zone users, therefore the population of interest was quite small. Secondly, it is often difficult to obtain acceptable response rates from business organizations. Therefore, to avoid using up this limited population and perhaps losing valuable data, pretesting with students appeared quite attractive. Essentially, three tasks were performed by the students: 1) learning about the concept of foreign trade zones as a type of facilitating service in international logistics systems; 2) playing the role of manager in filling out the questionnaire; and, 3) providing detailed feedback on the cover letter as well as on the research instrument. Prior to issuing the questionnaire instruments, the class had a brief discussion on foreign trade zone operations and their benefits. In addition, students were also given a two-page description of a foreign trade zone. Two days later, they were issued a packet containing a scenario, a cover letter, the questionnaire and a feedback sheet related to the questionnaire format and design. Each student received only one of the two scenarios. The first scenario described a hypothetical company which had been using a foreign trade zone, while the second company was considering whether to use a zone. Students assumed the role of a distribution manager who was asked to participate in this task. They were asked to respond to the questionnaire, using the information included in the scenario along with the previously issued foreign trade zone description handouts. Half of the class received the non-user scenario and questionnaire; the other half played the role of the manager of a zone-using firm. Students worked on this as an extra credit assignment outside the classroom, which yielded a response process more similar to the actual setting for managers. Two days later, 28 out of 32 students returned the questionnaire and the feedback sheet on how the instrument could be improved. A maximum of thirty possible points was allocated to this extra credit assignment. If a student only filled out the questionnaire, he or she received no more than five points. The feedback sheet and comments were worth up to the maximum of 25 points, depending upon their extent and depth. The instruction sheet also stressed what they would learn from this assignment. Specifically, they would better understand the foreign trade zone operations and their vital role in import and export processes. In addition, they would become more familiar with activities related to the primary data collection procedure used in the area of international marketing and logistics. The Questionnaire included questions concerning the firm’s characteristics, the manager’s attitude toward, and perception and knowledge of foreign trade zone operations. The measures consisted of categorical scales, five-point semantic differential scales, and absolute values (e.g. sales, dutiable inventory). An attempt was made to make the scenario sufficiently informative for students to respond to the questionnaire with knowledge instead of guessing. The two scenarios are shown in the appendix. Changes in the questionnaire itself were made based on the actual responses as well as on the detailed question-by-question feedback provided by the students. ASSESSMENT OF THE APPROACH One case is not sufficient for assessing the real value of this research/teaching method. However. this approach appeared to be successful in this particular situation. Clearly, from the authors experience, students who participated in the pretest grasped the concept and gained insight about factors important to international marketing and logistics decisions. It is an interesting and a more “real world” way of learning about the business environment. Students are also offered the opportunity to begin developing managerial decision making skills. At the same time, the researcher obtained very fruitful results on how to improve upon the questionnaire, both in terms of clarity and content. Limitations exist, however. The approach is very time consuming for the student and for the researcher. First, if students have no prior case method experience, they may be confused, discouraged and less likely to assist in the accomplishment of your goal. In such cases, the researcher may need to expose them to the case method prior to pretesting. Secondly, the instructor needs to spend extra time preparing the necessary information for students to respond knowledgeably to the instrument. This involves developing a more tailored lecture and writing up realistic scenarios. However, the researcher should be careful not to focus only on the relevant pieces of information or the task may become oversimplified. If this occurs, the task is no longer comparable to that of the managers and the role-playing exercise loses its original intent and value. Lastly, the opportunities for developing decision making skills may be lessened and research biases may result. Therefore, the greatest difficulty in implementing this approach is developing a student pretest that is both useful as a learning tool and useful to the researcher in evaluating questionnaire design. Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 237 CONDITIONS WHERE THE APPROACH IS MORE APPROPRIATE Due to the high degree of time and effort involved, this approach may have the best results in smaller, upper- division courses. It may also be more effective when the researcher is actually the instructor of the class so that he or she can build this assignment into the course schedule more naturally and more appropriately. In addition, students pretesting may be useful whenever the population of interest is limited in size or access and/or associated with lower response rates (e.g. business executive). In these situations, having students pretest the research instrument may lessen the chance of losing valuable respondents and data. SUMMARY This paper discussed the benefits of student pretests as an experiential learning exercise. The importance of pretesting is well documented in the marketing research literature. However, it is often conducted with students in a haphazard fashion where students do not receive substantial benefits from the pretesting experience. If planned properly, student pretesting can provide benefits to both researcher and student. A discussion and illustration of student pretesting was provided. Positive results were noted from implementation of this method and situations where student pretesting is most useful were indicated. APPENDIX Scenario A You are an international marketing manager for Chemscreen, a division of International Chemical Corporation. Chemscreen manufactures a line of industrial filters used in the chemical industry. Chemscreen’s annual sales last year were $22 million. It currently employs 245 employees. Most of the filters are manufactured in the U.S. Seventy percent of these products are sold in the U.S. while 30% are exported to South American countries and Canada. The production and materials cost of these filters could be cut by 15% (after deductions of shipping costs) if Chemscreen orders a component from Taiwan. This component is subject to a tariff rate of 20% of its imported price. A foreign freight forwarder advised you that it might be wise to source the component from Taiwan and bring them into the U.S. through a foreign trade zone. Its foreign subsidiary in Venezuela also felt that the use of an FTZ would simplify the customs procedure when the finished goods are distributed through South America. So the decision was made 3 years ago to use an FTZ in its distribution system. Since Chemscreen is a large-scale manufacturer, you decided to obtain permission to operate a subzone within the company’s plant in Tulsa, Oklahoma. The subzone is regarded as a part of the Tulsa FTZ #53. The annual volume of raw materials and components that were brought in from domestic sources was $1,000,000 and $1,500,000 in 1983 and 1984 respectively. Meanwhile, $500,000 and $700,000 worth of components were brought in from Taiwan during these last two years. Of all merchandise shipped out of the zone to the market, merchandise valued at $1,700,000 are shipped to domestic markets compared with $900,000 to South America and Canada. Scenario B Universal Electronics, Inc. (UEI) is a wholesaler for electronic equipment within the U.S. market. Some of the products are made domestically while others are imported. Its annual sales last year were $15,000,000 and the firm employed 178 people. Recently, UEI began to do more light assembling and repacking of its imported goods from Japan, South Korea, Singapore, and West Germany. The value of imports in 1984 was $1,000,000. Meanwhile, it wants to bring in some domestic-made products to be packed with imported items and distribute them to different customers throughout the United States. The company has been using a bonded warehouse to store its imports for several years. This bonded warehouse provides excellent warehousing facilities but it does not have enough space for any assembly. You, as the distribution manager responsible for the import function, are considering whether to use an FTZ. There are nearly 100 zones throughout the U.S. to choose from if UEI decides to use one. The larger markets are concentrated along the East Coast, while most of EDI’s imports come in through Los Angeles. REFERENCES [1] Cunningham, William H., W. Thomas Anderson Jr. and John H. Murphy (1974), “Are Students Real People?’ The Journal of Business, 47, 399-409. [2] Dillman, Don A. (1978), Mail and Telephone Surveys, New York: John Wiley and Sons. [3] Dunne, Patrick, Daulatram Lund and Vincent Luchsinger (1980), “The Use of Students as Surrogates: A Jeopardy Game?” in Marketing in the 80’s, R.P. Bagozzi, K.L. Bernhardt, P.S. Busch, D.W. Cravens, J.F. Hair and C.A. Scott, eds., Chicago: American Marketing Association, 312-315. [4] Lamb, Charles W., Jr., and Donald E. Stem, Jr. (1979), ‘An Evaluation of Students as Surrogates in Marketing Studies,” Advances in Consumer Research, J.C. Olson, ed., Ann Arbor, MI: Association for Consumer Research, 796-799. [5] Sewall, Murphy A. (1978), “Simulation and Experiential Learning as Human Subject Research, “ in Exploring Experiential Learning: Simulations and Experiential Exercises, D.C. Brenenstuhl and S.C. Certo, eds., 283-288. [6] Tybout, Alice M. and Gerald Zaltman (1974), “Ethics in Marketing Research: Their Practical Relevance,” Journal of Marketing Research, 11, 357-68. Table of Contents Volume 13, 1986 How to Use Microcomputers in Human Resource Management Courses What's My Line? An Exercise in Job Analysis, Description and Classification Relationships Between Team Cohesion Dimensions and Business Game Performance Four Factors Affecting Group Performance in Business Policy Simulations Enhancing Strategic Management Goal-Setting Skills in the Business Policy Course Using a Business Simulation in the Principles of Management Course - Learning Outcomes and Perceptions Personal Computers: Drexel's Experience Using Psychological Type to Enhance Negotiation Skills The Effects of Conflict Handling Styles on Negotiation Behavior: An Analysis of an Experiential Exercise An Experiential Test of Bargaining Decisions and Representation Self-Grading of Examination by Students, A Viable Alternative to Traditional Procedures Perceived Instructor Enthusiasm and Student Achievement An Experiential Approach to Teaching Subordinate-Oriented Communication Assessing the Effects of a Computerized Study Guide in Macroeconomic Principles: A Statistical Analysis A Sales Management Simulation for the PC an Integrative Tool for the Sales Management Course Teaching Re-evaluation of Salespersons Through the Use of a Simulation Game A Test of Student Performance and Attitudes and Performance Under Varying Game Conditions Developing and Testing Airways: A Marketing Simulation Experiential Learning Revisited: Some Thoughts on Designing More Adaptive Management Education Programs Developing the Competencies of Creativity and Accurate Self-Assessment Self-View and Norms: Some Longitudinal Laboratory Results Strategic Planning with an Experiential Case The ABSEL Research Consortium: Preliminary Steps and Position Papers An Analysis of ABSEL Conference papers (1974-1985) The Creation and Operation of a Data-base System for ABSEL Research Funding for a Research Consortium on Simulation and Experiential Learning Simulation - Indoctrination or Learning The Dilemma In Evaluating Classroom Innovations Personality Variables on Group Cohesion, Team Participation and Total Learning Justifications for and Problems in Developing and Using Computerized Experiential Activities Management Competency Models and the Life-Long Learning Project: What Role for ABSEL External Validity of Business Games Restructuring the University - An Experiential Exercise The Surrogate Class Exercise Images of Effectiveness: A Classroom Exercise The Algebran Industry: A Tutorial Framework Another Chance: A Multi-Leveled, Macro Business Game for Micro-Computers A Water Quality Management Simulation Game Acquisitions? Divestitures? Progress Report on a Conglomerate Game Enterprise: A Multi-Purpose Management Development Simulation Marketer: A Microcomputer Simulation in a High Tech Industry A Pedagogical Analysis of Alternative Tax Elections for Exploration Costs in the Mining Industry Using of Multiple Microcomputers Application Programs to Teach Fundamental Business Concepts and Practices Demonstrating the Role of Simulation in Strategic Operations Planning: A Case Study in Bank Check Processing Location Analysis The Subjective Side of the Decision Support System: A Pitfall for the Panacea A Group-Based Procedure for Revealing Judgmental Heuristics and Biases An Experiential Exercise in Bayesian Decision-Making Values for Selected Parameters in Physical Distribution Simulation and Games Computerized Business Management Simulations for Tyros The Management Decision Laboratory at New York University An Example: The Use of Management Games on Microcomputers by Computer Novices Enhancing Mainframe Simulation Via Microcomputers: Designing Decision Support Systems Integrating Personal Computers Into a Course as a Decision Support Tool A Decision Support System for Capital Funds Forecasting Performance and Attitudinal Affects of a Decision Support Package in a Business Game Using the Real World: Interviews with Woman Mangers The Cooperatively-Authored Student Handbook International Buying: An Experiential Exercise The Research/Teaching Interface: Turning a Pretest into an Experiential Exercise Building Microcomputer Business Simulations Simulation with Discrete and Continuous Mathematical Modeling An Interpolation Approach to Developing mathematical Functions for Business Simulations The Design of Simulation Models Using Thought Organizers The Assessment Center as a Teaching/Learning Device An Approach to Meeting the AACSB Guidelines for Introducing Skills and Personal Characteristics (SACs) Using an Interclass Simulating and Role-Playing Pedagogy Experiential Learning about the World of Work: A Program for Primary and Secondary School Educators Assessment of Relevant Personal Characteristics of Potential Entrepreneurs