PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS Classroom Response Systems in the Wild: Tech- nical and Non-Technical Observations http://dx.doi.org/10.3991/ijim.v8i1.3454 Jonas Vetterick, Martin Garbe, Andreas Dähn and Clemens H. Cap University of Rostock, Rostock, Germany Abstract—Classroom Response Systems (CRS) provide lec- turers a communication channel to get feedback from their students. In lessons with large audiences, CRS allow stu- dents to ask questions or state issues as the lesson continues. During the development and usage of our new CRS "Tweedback", we observed several technical and non- technical problems, which are likely to be general CRS is- sues. Observed problems are caused by necessary devices, their connectivity and lecturers’ and students’ different ways to use CRS. In this paper, we describe our observa- tions of technical and nontechnical problems and suggest solutions, which may be applied generically to interactive feedback systems. Index Terms—Classroom Response Systems, Audience Re- sponse Systems, Live Feedback, Large Audiences I. INTRODUCTION Smartphones, Tablets and Notebooks have become stu- dents’ everyday companions. They are used for communi- cation, entertainment, and assistance in various scenarios. Over the last years, researchers started to use these devices to create an additional communication channel between lecturers and students in lessons with large audiences. So called “Classroom Response Systems” (CRS) allow lec- turers to get feedback from their students as well as it al- lows students to get feedback from the lecturer while the lessons proceeds [3]. Whereas there are several types of feedback that can be given during a lecture, we use the following three types [18]. First, students may submit questions to both the au- dience and the lecturer. To indicate important questions an up voting mechanism may be implemented. We call this a Chatwall. Second, lecturers can challenge their students with quizzes, i.e. multiple choice questions. Third, stu- dents can provide feedback to lecturers by rating specific parameters of their speech, for example talking speed or volume. This publication provides an overview on approaches, existing projects and current solutions for live feedback in lessons with large audiences developed at German univer- sities. Whereas a live feedback application has to deal with many technical problems, nontechnical problems are often invisible – until they grow up and become an obsta- cle for users. Thus, we documented technical and non- technical problems we observed in CRS and suggest solu- tions to them. We hope that future and existing projects can use these results to improve their work. In general, we observed three main technical difficul- ties. First, the wireless communication infrastructure has to be available permanently and must carry the huge amount of data, which may occur in large lessons with more than 100 students. Second, lecturer’s screen stand and screen angle can become important issues to lecturers. Insufficient stands were very frustrating experiences for lecturers. Third, a CRS should be aware that all user input has to be reviewed prior to processing, as users may oth- erwise misuse the communication channel offered by the CRS. In addition to technical problems, we observed several nontechnical problems with CRS: Students want to reply to other students questions and they want to access a les- son's content even after the actual lesson ended. Further- more, young students seem to have a higher motivation to write comments, which are unrelated to the content of teaching. This behavior abates with the time of use, but lecturers should be aware of it. This paper provides solutions regarding these observa- tions and resulting issues. A fine-grained publication model provides students a communication channel which enables them to address only relevant recipients. Moreo- ver, a CRS integrated into the existing presentation appli- cation reduces the lecturer's distraction. The rest of the paper is organized as follows. Section II presents and compares existing projects. Section III de- scribes the technical observations we made regarding our CRS' use, whereas section IV covers nontechnical obser- vations. In section V we present our solutions to technical and non-technical problems. Section VI concludes the article with a summary and an outlook on future work. II. RELATED WORK The following sections give an overview on the most advanced projects and approaches, which allow at least one of the three live feedback types as previously defined. We compare several attributes of these projects (Table I). To categorize CRS we start by distinguishing between native and web applications, as devices not supported by native applications reduce the number of people, who can give feedback. Whereas native applications benefit from their underlying operating system and are able to use de- vice sensors or can be integrated into the system (and are often easier to maintain), web applications can be used on nearly every mobile device [7]. Furthermore, we take a look at different usage barriers. A question which reveals the asking person’s identity may build up a barrier, as students may hesitate to ask ques- tions perceived as embarrassing. Anonymity or pseudo- nymity may decrease students’ inhibition to provide feed- back [4], because anonymity and pseudonymity prevent students from identification by their lecturers [10]. iJIM ‒ Volume 8, Issue 1, January 2014 21 PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS TABLE I. COMPILATION OF CRS Another barrier of use is integration into the universi- ty’s software system. While an application can benefit from integration into existing university software system, it is often a problem to use this application without this software system [7]. Deeply integrated applications may exclude users, who are not members of this system, e.g. visiting lecturers.Finally we inspected projects for their implementation of the live feedback types Chatwall, Quiz and Speech Parameter.Kundisch et al. implemented Pingo, which aims to support the peer instruction concept for large classes [8] [14] [17]. By using a public accessible web application, it can be viewed on every modern browser, regardless of the used device. At the moment, lecturers have to create an account in order to use it, whereas students can access lessons without creating an account. Pingo features only quizzes and is not integrated into any existing presentation software. A great advantage of Pingo is its capability to create questions in advance. Moreover it provides an overview on all created and used lessons. According to their statistics7, Pingo is well ac- cepted (more than 1700 created lessons between October 2012 and November 2013). Feiten et al. developed Smile [1] [2], an application that consists of two different parts. First, Smile provides a Windows application for lecturers. Second, students can access Smile using a web application, an app on iOS, or an app on Android. The lecturer’s application is used to create and control lessons, so lecturers are able to create their Quiz questions in advance. Furthermore, Smile al- lows students to ask and answer questions, which may be used as replacement for a Chatwall. Moreover, students can state their general understanding: They can send a signal whenever they do not comprehend the lecturer. Smile has a highly adjustable user interface, featuring many settings. The developers found an elegant way to combine the Quiz feature with the lecturer's slides pre- sented in a third party application: By overlaying the slides with a small transparent window containing the 1 https://arsnova.eu 2 http://invote.de 3 http://mytu.tu-freiberg.de 4 https://pingo.upb.de 5 http://tweedback.de 6 http://www.smile.informatik.uni-freiburg.de 7 http://pingo.upb.de/stats current quiz’ results, the lecturer is able to integrate the quiz into his presentation easily. Whereas this user inter- face allows lecturers to configure the application for their needs, Weber et al. state that the usability and responsibil- ity of Smile does not yet fit students’ needs [19]. Gommlich et al. extended their university’s personal learning environment myTU with a functionality, which allows students to rate the speed of the lecturer’s presenta- tion [6]. It also features a generic “Stop” button, which may be used whenever students have an issue regarding the lecture or the lecturer’s presentation. Moreover, stu- dents may ask short questions, which are immediately shown on the lecturer’s device. The myTU-App can be considered more an assistant for student’s life than a pure CRS: It is used to manage students class schedules, their lending from the library, the cafeteria plan and provides many other tools supporting students’ everyday life. Be- cause this application has been developed for the TU Freiberg, it is deeply integrated with the university’s soft- ware system and every user needs a university account to use the app on Android or iOS. ARSNova is a CRS implemented as a web application. It may be used anonymously or after authentication with a university, Facebook, or Google account. It is possible to integrate ARSNova into different learning management systems such as Moodle, Ilias, or StudIP. It provides func- tionality to create real-time quizzes and to rate a lecturers Speech Parameters. Furthermore, a presentation view sim- ilar to Smile allows lecturers to overlay the presentation slides with the current quiz’ results [12]. Tweedback implements all three previously described features, and is provided as a public accessible web appli- cation. Therefore, no accounts are needed to create or par- ticipate in a lesson. Tweedback provides pseudonymity for students and lecturers and is not integrated in the uni- versities software system [5]. Currently it is subject to a refactoring process. Subsequently to this process, Tweed- back will be released under an open source license inVote is a project developed by Netzmanufaktur GmbH for TU Dresden. It serves real-time quizzes and is implemented as a web application. Lectures have to create an account prior to starting a quiz, whereas students can answer anonymously and without an account to these quizzes [16]. ARSNova1 inVote2 myTU3 Pingo4 Tweedback5 Smile6 Native or Web App Web app Web app Native Web app Web app native for lectures & web app for students Needs Authentica- tion No Only Teachers Yes Only Teachers No Yes Anonymity or Pseu- donymity Anonymity Anonymity Anonymity Anonymity Pseudonymity Anonymity Open Source Yes (GPL) No No No (Planned) No (Planned) No Single App or Inte- grated Both Integrated Both Single app Single app Single app Overlay Functional- ity Yes No No No No Yes Chatwall No No Yes No Yes Yes Quiz Yes Yes No Yes Yes Yes Speech Parameter Yes Yes Yes No yes Yes 22 http://www.i-jim.org PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS Table I compares the discussed projects. It provides all properties and features considered important. It seems remarkable, that all presented projects provide anonymity or pseudonymity. Only Tweedback and Smile provide all three kinds of live feedback, whereas ARSNova and Smile stand out with their overlay functionality, which provides information directly on a lecturers presenting screen. III. TECHNICAL OBSERVATIONS Due to the wide spread presence of smartphones, it seemed plausible to use a purely web-based architecture. Most of our students own a smartphone, with which they can use the Universities Wi-Fi network to connect to the Internet. Moreover, GSM and UMTS connections can also be used. In contrast to CRS hardware solutions, e.g. “Qwizdom” [13] and “Powervote” [11], which build up their own communication infrastructure, web-based archi- tectures use existing Wi-Fi infrastructure. Therefore, they also share problems, such as connectivity problems caused e.g. by massive peer-to-peer file sharing. Our observations regarding communication infrastructure, input devices, and software requirements are described in the following section. A. Communication Infrastructure Availability Prior to using a web-based CRS, Wi-Fi has to be avail- able in lecture halls. Not only the Wi-Fi availability but also the capacity has to be checked. It is important to know the exact location of installed Access Points (AP) and which area they cover. It is difficult to make estima- tions about upper bounds of the number of users in a Wi- Fi network. This number depends highly on the users’ behavior, installed hardware, and the web application. Consulting the corresponding IT administration and using their experience is one way to estimate the maximum number of users which an AP can handle in practice. B. Input Device The process of introducing a web-based CRS to lectur- ers consists of the following steps: • Verification of communication infrastructure • Learn how to use the CRS • If necessary: Introduce additional display/input de- vice Lowering the initial barrier is important for a wide ac- ceptance. Therefore, offering support for lecturers new to a CRS can help accomplishing the tasks mentioned above. Depending on the CRS’ capabilities, an additional dis- play/input device is needed. Only a small number of CRS, namely ARSNova and Smile, have some sort of overlay window for presentation slides. Anyhow, this overlay shows only a small amount of information. For example, it is possible to show a smiling face if users are satisfied with the current talk or a sad smiley if it is not understand- able. For features showing more information content, such as a Chatwall, a small overlay is not feasible. To keep an overview on new content, it is important to avoid inter- rupting the teaching process. Therefore switching between presentation and CRS window on one device cannot be recommended. One solution is using additional display devices, e.g. tablet computers. When mounting the tablet, horizontal and vertical angle are important to assure an optimal view on the screen. Many stands for tablets are designed to place the tablet in a fixed place for watching films or reading a book. Conse- quently, many stands are insufficient for a CRS scenario, in which lecturers interact with devices. Lecturers scroll a webpage, activate features and highlight questions. Many stands are not designed for pushing devices at the upper corners. This physical problem has – regardless of how trivial it may seem – to be respected when choosing a stand, as this detail has significant influence on the lectur- ers’ experience with a CRS. C. Platform During the design of Tweedback only very few assump- tions regarding the user’s platform were made. The only requirement was a browser application and integrated Wi- Fi. This requirement is fulfilled by almost all modern tab- lets. Therefore, building a web application with user inter- face frameworks, such as the Bootstrap framework, and the use of standard technologies (HTML, JS, and CSS), minimizes restrictions on end-user devices. On the server side, we were able to verify the experi- ence with users the Pingo project [17] made. As we are working with pseudonyms, every user has a nickname such as “user123” or “user211”. In early versions of Tweedback this information was saved in a cookie. First tests with Computer Science students lead to massive ma- nipulation of usernames. By changing the cookie’s con- tent, students changed their nicknames. Therefore it is highly recommended to treat all transmitted data send by users as possible manipulation attempts. IV. NONTECHNICAL OBSERVATIONS Further observations deal with user behavior and new usage scenarios for lecturers. A Chatwall feature can have a restricted functionality by design. One possibility is to allow only new questions and to deny responses. This way, users can only ask ques- tions and lecturers have to answer them. The idea of providing a restricted usage is also supported by Cognitive Theory findings [9][15] which states that working memory of human cognitive system consists of channels which can be overloaded. Too many pictures or too much text overloads these channels. Too much unstructured information has the same effect. As a consequence, we decided not to allow users to chat with each other or to comment on other posts. Contrary to our intention, users began to use our system for communication with each other anyway. They replied to other posts by prefixing new posts with terms such as “@user123”. This behavior shows that there is a need for a reply function which contradicts with recommendations of didactic researchers. In lessons with high activity, the Chatwall has to serve several needs: On the one hand, users want to see the lat- est posts and vote on them. On the other hand, users want to see the “hot topics”, i.e. those questions with most votes. Providing different sorting of questions turned out to be important for lecturers. In general, lecturers use a “most voted” view of the Chatwall, as this view empha- sizes questions and problems addressed by most students and thus probably important. If new questions are the lec- turer’s focus, the “latest posts” view can be used. Any- iJIM ‒ Volume 8, Issue 1, January 2014 23 PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS how, in case of a large audience with many students who ask many questions, it can be hard to overview all new messages in short time. We were able to distinguish between usage patterns, which seem related to the duration of CRS usage. We identified an initial phase concerning students who used Tweedback for the very first time. They began to “play” with it, creating lots of small messages without lecture- related content. The initial phase is followed by a produc- tive phase, in which the amount of such messages drasti- cally reduces. The productive phase lasted for the rest of the term. From the lecturers’ view, especially in the initial phase many messages can be classified as spam, as they do not address a problem related to the current lecture. This behavior was expected and after some lectures, we observed a normal behavior where students asked only serious questions and used the system as intended. We also experienced a demand to access all results after the lesson ended. Lecturers wanted to evaluate and ana- lyze the data that has been generated by the students to learn and understand their students’ problems. Students on the other hand wanted to share these emerged problems in order to get a deeper knowledge in specific topics, slides, formulas, definitions, or simply to solve issues that are not directly related to the actual lesson’s subject. For this rea- son, durable access to the lessons results turned out to be very important. In addition to the ability to have durable access to a les- son’s data, some lectures demand a functionality to ask questions before the lesson starts. Lecturers may use such a feature to check their students’ knowledge of the previ- ous lessons or to ask questions which may give an over- view on the students’ motivation (“Which semester are you?”, “Which study path are you associated?”, …). Be- cause students are not entering the lesson simultaneously, it may be useful to ask such questions in a loop before the lesson starts. V. SOLUTIONS Our observations showed that users want to comment on questions and find themselves a way to override the system’s restrictions regarding replies on other comments. Keeping this experience in mind, we will implement a reply-feature for the Chatwall. In the first place, this seems to contradict with the didactic researchers’ recom- mendation to not create another communication channel between students. But the problem is that there already exist other communication channels once the smartphone is up and running within a lesson. All apps on smartphones which allow communicating with others pre- sent a form of parallel communication channel. Therefore, our aim is to fulfill students’ need and implement a reply function as it may keep them in the CRS. Because there are already other communication channels available, we believe that uses will not be more distracted than before. In case of many replies to a question,, the Chatwall page will become very long. Additionally, this will make it more difficult to keep track of current questions in short time. Therefore, we suggest different views on the Chat- wall. In the following, we will differentiate between three motivations of a reply and explain their resulting views. First, private replies (which are exchanged between two users) will reduce messages perceived as “noise” by other users. For a better overview, these messages are not shown to other users than the regarding recipient since others were not addressed explicitly. Second, semi-private (audience only) replies reduce messages perceived as “spam” by lecturers. We observed a relevant amount of questions which are directed to the audience, e.g. organization of collective lunch. Lecturers perceive these messages as spam. Allowing for semi- private messages lets the audience communicate without disturbing teachers. A possible third kind of replies are public comments. Such comments are visible to all users who have joined a lecture. Separating these three reply motivations and creat- ing a unique view for each of them enables users to com- municate without disturbing those not interested in the communication. Figure 1 summarizes all three motiva- tions and their covering area in an auditorium. Figure 1. Shows a lecturer and the auditorium as well as the communi- cation spaces of public, semi-private (audience) and private communi- cations Different needs for sorting, as stated above, can be im- plemented in a simple way. Depending on their needs, users can switch between two views of Question Wall. The view “voted most” will be primarily used by teachers. This way they keep an overview of questions many people voted for and they are probably important. The second view “latest questions” aims for students, as they can rec- ognize new questions without wasting time to scroll down. Up to now, we recommend using an additional tablet device when using a CRS in a lesson. This measure eases interaction with the Chatwall. This contradicts with the obvious interest of lecturers who want to carry as few de- vices as possible. To solve this problem, we plan to incor- porate features into presentation software such as Mi- crosoft PowerPoint. Once implemented, lecturers will not need to switch between devices or applications in case of Quiz and Speech Parameters. There exist other solutions, e.g. to incorporate the Chatwall into presentation slides on audience devices. Our vision is that students can write their comments live di- rectly into the slides. This would be a great advantage, as many questions refer to specific parts of a presentation. If students could leave comments directly in the slides, e.g. as annotation, the problem would be directly related to its origin. We imagine a web-based solution for this scenario, where presentation slides are available online. Users can view the slides simultaneously on the presenter’s projec- tion as well as on their own device. Furthermore, students 24 http://www.i-jim.org PAPER CLASSROOM RESPONSE SYSTEMS IN THE WILD: TECHNICAL AND NON-TECHNICAL OBSERVATIONS may highlight or comment interesting parts visible to oth- er students. Concerning the feature of durable access to a lesson’s user generated content, we implemented an interface showing all this information. Therefore we store and dis- play all quizzes and all Chatwall posts. The Speech Pa- rameter will be shown as a line graph, where each line represents the ratings of a specific parameter’s develop- ment during the lesson. VI. SUMMARY Using students’ smartphones to access a CRS has many advantages. To use a web based CRS, a communication infrastructure, such as Wi-Fi network, is necessary. Getting an overview of the audience’s feedback in short time is very important to lecturers. A Chatwall can contain many questions. Therefore, we recommend an external device (e.g. a tablet computer) to present this additional information to lecturers. We observed that students use the communication channel in lessons to reply on questions – in groups or personally. Thus, we suggest providing private, semi- private (audience only) and public conversations. This allows users to comment on other users’ posts and pre- serves a low spam level. It has to be evaluated whether this new possibility of commenting is distracting users significantly more than the existence of other communica- tion channels on the smartphone. Another observation were repeated requests for differ- ent sortings regarding the Chatwall. We identified two different sorting methods: ‘by date’ and ‘by upvotes’. Additionally, we observed the need for durable access to a lesson’s results. This feature should be implemented in a read-only view, where all Chatwall posts and quizzes are listed. For Speech Parameters, a line graph showing ratings of each parameter over a lesson’s duration seems a promising visualization. Future research should focus on a wider application for this real-time communication channel between audience and lecturer. This may cover different types of live feed- back, such as comments and annotations directly in the lecturer’s slides. Furthermore, there is need for long-term didactical evaluations in CRS with large audiences. REFERENCES [1] L. Feiten, M. Buehrer, S. Sester and B. Becker, "SMILE- Smartphones in Lectures-Initiating a Smartphone-based Audience Response System as a Student Project," in 4th International Con- ference on Computer Supported Education (CSEDU2012), 2012, pp. 288-293. [2] L. Feiten, K. Weber and B. Becker, "SMILE: Smartphones in der Lehre – ein Rück- und Überblick," in GI2013, 43. GI- Jahrestagung, Workshop iLearn, 2013. [3] C. Fies and J. Marshall, "Classroom response systems: A review of the literature," in Journal of Science Education and Technolo- gy, vol. 15(1), Springer, 2006, pp. 101-109. [4] M. Freeman and A. Bamford, "Student choice of anonymity for learner identity in online learning discussion forums," in Interna- tional Journal on E-learning 3.3, 2004, pp. 45-53. [5] M. Garbe, J. Vetterick and C. H. Cap, "Tweedback: Online Feed- back System for Large Lectures," in GI2013, 43. GI- Jahrestagung, Workshop iLearn, 2013. [6] F. Gommlich and G. Heyne, "Persönliche Lernumgebung - Archi- tektur für Smartphones," in GI2013, 43. GI-Jahrestagung, Work- shop iLearn, 2013. [7] W. Jobe, "Native Apps Vs. Mobile Web Apps," in International Journal of Interactive Mobile Technologies, Vol. 7(4), 2013. [8] D. Kundisch, M. Sievers, A. Zoyke, P. Herrmann, M. Whittaker, M. Beutner, G. Fels and J. Magenheim, "Designing a web-based application to support peer instruction for very large groups," in International Conference on Information Systems (ICIS2012), 2012. [9] S. L. Muthkumar, "Creating interactive multimedia-based educa- tional courseware: cognition in learning," in Cognition, Technolo- gy and Work, Vol. 7, 2005 pp. 46-50. [10] A. Pfitzmann and M. Köhntopp, "Anonymity, unobservability, and pseudonymity—a proposal for terminology," in Designing privacy enhancing technologies, Springer Berlin Heidelberg, 2001. [11] Powervote, "Powervote: http://www.powervote.com [23.01.2013]," 2013. [12] K. Quibeldey-Cirkel, C. Thelen, P.-C- Volkmer, D. Gerhardt, D. Knapp and J. Kammer, "ARSnova: ein Audience Response Sy- stem für Inverted-Classroom-Szenarien mit Unterstützung von Just-in-Time Teaching und Peer Instruction," in DeLFI 2013--Die 11. e-Learning Fachtagung Informatik, Gesellschaft für Informa- tik, 2013. [13] Qwizdom Ltd, "Qwizdom Q5: http://www.qwizdom.co.uk/q5.php [23.01.2013]," 2012. [14] W. Reinhardt, M. Sievers, J. Magenheim, D. Kundisch, P. Herrmann, M. Beutner and A. Zoyke, "PINGO: peer instruction for very large groups," in 21st Century Learning for 21st Century Skill, 2012, pp. 507-512. [15] W. R. Robinson, "Cognitive theory and the design of multimedia instruction," in Journal of Chemical Education, Vol 81, 2004, pp. 10-13. http://dx.doi.org/10.1021/ed081p10 [16] L. Schlenker and S. Beyer, "Online In Der Vorlesung-Potentiale Digitaler Medien Für Aktives Lernen," in 11. Workshop on e- Learning (WeL2013), 2013. [17] M. Sievers, W. Reinhardt, D. Kundisch and P. Herrmann, "Devel- oping Electronic Classroom Response Apps for a Wide Variety of Mobile Devices: Lessons Learned from the PINGO project," in 11th World Conference on Mobile and Contextual Learning (mLearn2012), 2012, pp. 248-251. [18] J. Vetterick, M. Garbe and C. H. Cap, "Tweedback: A Live Feed- back System for Large Audiences," in 5th International Confer- ence on Computer Supported Education (CSEDU2013), 2013. [19] K. Weber and B. Becker, "Formative Evaluation des mobilen Classroom-Response-Systems SMILE," In E-Learning zwischen Vision und Alltag (GMW2013 eLearning), 2013. AUTHORS Jonas Vetterick is with the Department of Computer Science, University of Rostock, Germany (e-mail: jo- nas.vetterick@uni-rostock.de). Martin Garbe is with the Department of Computer Science, University of Rostock, Germany (e-mail: mar- tin.garbe@uni-rostock.de). Andreas Dähn is with the Department of Computer Science, University of Rostock, Germany (e-mail: andre- as.daehn@uni-rostock.de). Clemens H. Cap is with the Department of Computer Science, University of Rostock, Germany (e-mail: clem- ens.cap@uni-rostock.de). This article is an extended and modified version of a paper presented at the 5th International Conference on Computer Supported Education (CSEDU 2013), 6-8 May 2013, Aachen, Germany.Manuscript received 01 December 2013. This work was supported in part by the BMBF and the University of Rostock. Submitted 01 December 2013. Published as re-submitted by the authors 05 January 2014. iJIM ‒ Volume 8, Issue 1, January 2014 25 iJIM Vol. 8, No. 1, January 2014 Classroom Response Systems in the Wild: Technical and Non-Technical Observations