International Journal of Interactive Mobile Technologies (iJIM) – eISSN: 1865-7923 – Vol. 14, No. 18, 2020 Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Chatbots for Brand Representation in Comparison with Traditional Websites https://doi.org/10.3991/ijim.v14i18.13433 Johannes Kühnel, Markus Ebner, Martin Ebner () Graz University of Technology, Graz, Austria martin.ebner@tugraz.at Abstract—Chatbots have gained enormous popularity in recent years. IT gi- ants such as Microsoft, Google and Facebook have taken an interest in automated conversations. Messaging apps like WhatsApp and Facebook Messenger are playing an increasingly important role in smartphone usage and communication in general perfect conditions for chatbots. This paper provides an introduction of recent chatbot development in general and in customer environments. As part of this work, a chatbot was developed for the Austrian IT company CodeFluegel GmbH. The chatbot, named Theodore, provides information about CodeFluegel via Facebook Messenger and webchat, like the existing company website. The design process and implementation of the chatbot as well as architectural consid- erations are explained throughout this document. In a user study, participants per- form typical tasks with the website and the chatbot. The usage of both platforms is evaluated in order to identify advantages and disadvantages as well as differ- ences compared to the other technology and to draw conclusions. Findings to this study indicate promising prospects for chatbots as alternative touchpoints for cus- tomers and others and a way to replace and enhance traditional websites. Keywords—Chatbots, messenger, websites, natural language understanding, brand representation 1 Introduction Chatbots and conversational interfaces have become increasingly popular in recent years. At the developer conference Build 2016, Microsoft’s CEO Satya Nadella boldly proclaimed that “Bots are the new apps” [1]. Several factors have nurtured this trend. On the one hand, messaging applications like WhatsApp and Facebook Messenger are attracting billions of users (see Table 1) and occupying a large portion of a user’s screen time [2]. On the other hand, major technology companies like IBM, Microsoft, Google, Facebook and Amazon have taken an interest in providing development tools and nat- ural language processing centered around chatbots, resulting in easier development and wider acceptance and adaption of conversational interfaces. [3] Since the invention of the Internet, companies have been using websites as means to get in touch with potential customers, employees and generally interested parties. This paper aims to evaluate the feasibility and usability of chatbots in such a context, given 18 http://www.i-jim.org https://doi.org/10.3991/ijim.v14i18.13433 mailto:martin.ebner@tugraz.at Paper—Chatbots for Brand Representation in Comparison with Traditional Websites the immense popularity of previously mentioned messenger and social media platforms. For this evaluation, a chatbot was developed for the Austrian IT company CodeFlügel GmbH1. The chatbot provides information about CodeFlügel via Facebook Messenger and webchat, similar to the existing company website. The design process as well as architectural considerations and details of the implementation are explained throughout this document. For the evaluation itself, a user study is conducted where participants perform typical tasks with the website and the chatbot. The usage of both platforms is evaluated in order to identify advantages and disadvantages as well as differences compared to the other technology. Based on these evaluations, conclusions will further be drawn concerning, for example, differences in speed and user satisfaction. Findings to this study indicate promising prospects for chatbots as alternative touchpoints for customers and others and a way to replace and enhance traditional websites. Table 1. Monthly active users of popular messaging platforms with bot support according to [4], [5] and Telegram2. Messaging Platform Monthly Active Users (in millions) WhatsApp 1,500 Facebook Messenger 1,300 Weixin / Wechat 1,083 Twitter 326 Skype 300 Telegram 200 LINE 194 2 Related Work Chatbots provide an advantage compared to traditional mobile apps. Apps are prone to “app fatigue”, which describes the tiredness of users to install new apps. Other driv- ing factors of bots are the growth of messaging apps and social media, as well as the support of large companies like Facebook and Microsoft in the recent years [3] [6]. With these factors in mind, this chapter focuses on some case studies around the devel- opment and research of chatbots and conversational interfaces. Shawar et al. showed a chatbot answering Frequently Asked Questions (FAQ) for the School of Computing3 (University of Leeds). The bot used pre-processed data of the school's online FAQ on their website and pattern matching to map user input to ques- tions and retrieve the correct answers. Users “found it a novel and interesting way to access the FAQ using natural language questions”. Compared to a Google search, users were able to find a relevant answer more often and the majority of them preferred using the chatbot. Driving factors were the chatbots ability to provide direct answers to the 1 https://www.codefluegel.com, visited on 2020-01-02 2 https://telegram.org/blog/200-million, visited on 2019-02-13 3 https://engineering.leeds.ac.uk/computing, visited on 2019-03-14 iJIM ‒ Vol. 14, No. 18, 2020 19 https://www.codefluegel.com/ https://telegram.org/blog/200-million https://engineering.leeds.ac.uk/computing Paper—Chatbots for Brand Representation in Comparison with Traditional Websites question if there was only one matching answer and fewer links in general, resulting in less search time and better overview. [7] Almost 50% of Internet users in the U.S. use social media to contact customer ser- vice. According to recent studies, more than two thirds of users expect an answer within an hour, but it usually takes over six times as long to get a response. Scaling can be a problem with high numbers of requests, which chatbots could solve. Xu et al. built a chatbot for customer service on Twitter, as a feasibility study on such software. They used deep learning algorithms and trained their bot with nearly a million Twitter con- versations from more than 60 different companies and brands. One of their key findings was that more than 40% of customer requests are only emotional and not looking for any kind of information or help, possibly due to the nature of social media. Their cus- tomer service bot showed results close to a human when handling such emotional re- quests, while informational requests seem to require more complex training. [8] Fig. 1. Chart showing potential annual savings of US salaries in billion US dollars by using chatbots. Source: Business Insider4 Another example for bots in customer interaction is WAH Nails5. Sharmadean Reid, the founder of the company, said that 30% of reservations were already handled by their chatbot only two months after its launch and plans to enhance it further using AI were already underway [9]. While this is only a single example, chatbots have the potential for significant savings in such applications, where human agents could be replaced, or new services created. According to an infographic (see Figure 1) by business magazine 4 https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12, visited on 2019-03-20 5 https://wah-london.com/, visited on 2019-03-14 20 http://www.i-jim.org https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12 https://wah-london.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Business Insider6, potential annual savings in customer service in the United States of America alone amount to $23 billion US dollars7. Beriault-Poirier, Tep, and Sénécal conducted an exploratory user study similar to the one shown in this paper. The goal of the study was to learn more about the user experi- ence provided by chatbots from different brands on Facebook Messenger, namely food and beverage company Whole Foods8, clothing line Tommy Hilfiger9 and travel search site Skyscanner10. Users were asked to complete two tasks for each brand (six in total) one on their website and one with their Facebook Messenger chatbot. Chatbots and websites were then evaluated from 1 to 7 in terms of usefulness, ease of use, ease of learning and satisfaction. User experience scores and abandonment rates were favour- ing the companies' websites. Analysis of the participants' facial expressions, however, showed that chatbots generated more positive emotions. Participants liked the chatbots' ability to deliver quick and precise answers and also showed a willingness to use chat- bots in the future. [10] Chatbots for customer interaction could also impose new problems. Heckmann and Kraus mention that depending on the specific use case, personal data must be handled carefully, and usage of such data has to be disclosed. Commercial usage could also come with the requirement to inform users about those commercial intentions – for example, disclosing advertisements. Especially with the recent introduction of the Gen- eral Data Protection Regulation (GDPR)11, companies must take special care of pro- tecting personal data in and outside of the European Union (EU). [11] While bots can help in building a brand and customer relation, a bot's personality is also able to do harm. As seen with Microsoft's Twitter-bot Tay (see [12]), machine learning and AI could lead to unforeseen behaviour, which in turn could potentially damage a brand or company [13]. Brands and developers also have to keep in mind that a chatbot's presentation and persona can result in unwanted perception and emotions, especially in regard to the uncanny valley effect12 [14] [15]. 6 https://www.businessinsider.de, visited on 2019-03-20 7 https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12, visited on 2019-03-20 8 https://www.wholefoodsmarket.com, visited on 2019-03-19 9 http://www.tommy.com/, visited on 2019-03-19 10 https://www.skyscanner.com/, visited on 2019-03-19 11 https://ec.europa.eu/info/law/law-topic/data-protection_en, visited on 2019-03-17 12 Described as “the feeling of eeriness and discomfort towards a given medium or technology that frequently appears in various kinds of human–machine interactions” by [17]. iJIM ‒ Vol. 14, No. 18, 2020 21 https://www.businessinsider.de/ https://www.businessinsider.de/80-of-businesses-want-chatbots-by-2020-2016-12 https://www.wholefoodsmarket.com/ http://www.tommy.com/ https://www.skyscanner.com/ https://ec.europa.eu/info/law/law-topic/data-protection_en Paper—Chatbots for Brand Representation in Comparison with Traditional Websites 3 Theodore, A Chatbot As part of this case study, a chatbot called Theodore was created. The chatbot's main use is to represent a company, namely the Austrian-based software developer CodeFlügel GmbH. CodeFlügel is located in Graz, Austria and specializes in develop- ing augmented reality mobile applications. Virtual reality, traditional apps, web apps and custom projects are also part of their offered services. CodeFlügel has about twenty employees consisting of software developers, sales staff and other people from the op- erative business such as a human resources manager, marketing personnel and the chief executive officer. Like most companies, CodeFlügel uses a website13 to represent themselves on the Internet and inform potential interested parties about their services and other kinds of information. Content and features of the website include, but are not limited to, infor- mation about what CodeFlügel offers, a blog and a list of open positions. CodeFlügel also engages in social media platforms, one of which is Facebook14. Fa- cebook offers the possibility to directly contact a company via their own Facebook Messenger15. Messenger is a chat and messaging platform providing APIs to build third-party tools and more importantly chatbots. Theodore was designed to reproduce most of the website's features and to work with Facebook Messenger as well as a custom webchat, which could be embedded on a web- site. Existing APIs and content should be (re-)used as much as possible. For example, if a user asks the chatbot “What does your company do?”, it should answer with a proper explanation about the company’s products and services. 3.1 Target audience In order to design a company chatbot, it is imperative to recognize and understand the target audience of CodeFlügel. The website users can be divided into three groups: Clients: A client or potential client is someone interested in purchasing a product or service from the company typically a mobile application or a website. Some of them might have a very specific request or idea, while others are interested in consultation and guidance. They most likely want to know what the company is offering, their level of expertise in specific technologies and how to contact them. The technological exper- tise of the customers varies greatly and ranges from beginners to experts. For clients, the chatbot would not only offer information, but also function as a demonstration of the company's previous work on chatbots. 13 https://www.codefluegel.com, visited on 2018-10-28 14 https://www.facebook.com, visited on 2019-01-20 15 https://developers.facebook.com/docs/messenger-platform, visited on 2019-01-20 22 http://www.i-jim.org https://www.codefluegel.com/ https://www.facebook.com/ https://developers.facebook.com/docs/messenger-platform Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Potential employees: Potential employees are usually students with little job expe- rience, but a technological background or less likely people interested in marketing, sales or human resources. They could use the website or chatbot for general information about the company or to inform themselves about job openings or projects the company has worked on. Blog readers and everyone else: This group of users is probably not looking for anything specific about the company, but rather for one of the varying blog posts or general information about the technology used at the company. They most likely found the website / chatbot via a search engine such as Google16 or a social media channel such as Facebook. The chatbot provides little benefit for such users, besides interacting with the user and possibly generating interest and publicity. 3.2 Dialog design With such a target audience in mind, the chatbot's requirements and a feature set were defined in consultation with CodeFlügel. The chatbot should feature almost all of the website's content. Since the interaction is different from browsing a website, some additional design decisions have to be made. For example, a help function and a fallback dialog help to enhance the user experience. User input is limited to text input and quick reply buttons, while the output can incorporate media and specific layouts as well. Greeting: Theodore must be able to greet users in order to help them understand that they are chatting with the company chatbot of CodeFlügel. Thus, the chatbot first in- troduces himself and then shows some examples about what it can help the user with. About the company: The chatbot should be able to provide a short explanation about what CodeFlügel does and which services they offer, given an input like “What does CodeFlügel do?” (German: “Was macht CodeFlügel eigentlich?”). About products and service: Products and services should be explained by Theo- dore. Furthermore, the chatbot should provide information about completed projects. Contact information: The most important contact information should be displayed by the chatbot. This includes the office address, phone number and email address. A link to Google Maps17 is provided to help users in locating the office. Open positions: Theodore should inform users about open positions and provide a way to contact CodeFlügel. In case of no vacancies, an information about speculative applications and where to apply is shown. Newsletter subscription: CodeFlügel uses Mailchimp18 as a service to handle their newsletter. The chatbot should provide a way to subscribe to the newsletter by access- ing the Mailchimp API19. 16 https://www.google.com, visited on 2019-01-20 17 https://maps.google.com, visited on 2019-01-20 18 https://mailchimp.com, visited on 2019-01-20 19 https://developer.mailchimp.com, visited on 2019-07-03 iJIM ‒ Vol. 14, No. 18, 2020 23 https://www.google.com/ https://maps.google.com/ https://mailchimp.com/ https://developer.mailchimp.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Blog: Blog posts are published regularly by CodeFlügel. The website uses Word- press20 as its content management system. Theodore should output the most recent blog posts by accessing the website's REST API21. Social media links: CodeFlügel's social media handles should be shown and linked to their respective platforms, if a user asks for this information. This includes Facebook, Instagram22 and Twitter23. Help: Another important aspect in designing a chatbot is to provide help, if a user gets stuck or does not know what to do. Consequently, if a user asks for help, a short description about the chatbot's abilities should be shown. Fallback: The chatbot is made for a specific domain which defines the capabilities. If a user input is outside of this domain or the Natural Language Understanding (NLU) engine is unable to detect the user's intent, some kind of fallback should be triggered. For example, Theodore is not designed to announce the weather. Thus, if asked whether it is going to rain today, the chatbot should tell the user that he could not understand what the user wants or that it does not know an answer to the specific request. If this happens several times, the chatbot should present a way to get in contact with a real person. Theodore should feature informal language and a friendly attitude. The chatbot should make clear that it is a program and not pretend to be a real person, which should help to avoid the uncanny valley effect mentioned in Related Work. 3.3 Architecture and implementation Since CodeFlügel maintains a Facebook page as well as a website, the chatbot should be usable on both platforms. Facebook Messenger is used for the first, while a custom webchat is used for the second. Messenger already provides an existing messaging plat- form and chat experience. Only the backend needs to be developed for this platform. A webchat, however, also needs its own User Interface (UI) and ways to communicate with the backend. The same backend is used for both platforms. Facebook provides an API to com- municate with Messenger as well as webhooks to handle incoming messages and ac- tions. The webchat uses web-sockets to offer a fast and responsive chat experience. In order to streamline the development process and to increase the maintainability of the chatbot, Facebook Messenger's message format (in JavaScript Object Notation, short JSON) is used for both platforms and its UI components are recreated for the webchat. These components include simple text, Messenger's Generic, List, Button and Media Templates. 20 https://wordpress.org, visited on 2019-01-20 21 https://developer.wordpress.org/rest-api, visited on 2018-04-16 22 https://www.instagram.com, visited on 2019-01-20 23 https://twitter.com, visited on 2019-01-20 24 http://www.i-jim.org https://wordpress.org/ https://developer.wordpress.org/rest-api https://www.instagram.com/ https://twitter.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Typing indicators to simulate the behaviour of typing messages and quick reply but- tons (see Figure 2) were implemented as well. This also makes the different platforms more consistent and increases usability as users are presented with a familiar chat ex- perience. Instead of using a bot development framework to start the development, a custom implementation was chosen. While such frameworks provide functions and templates for quickly achieving first results, they tend to only support subsets of the features of- fered by some bot platforms. Fig. 2. Implementations of Quick Reply buttons (labelled “Ja, abonnieren”, “Nicht abonni- eren”) on the webchat (left) and Facebook Messenger (right). Google's Dialogflow24 (previously known as API.AI) was chosen as the NLU ser- vice because it shows promising development with features such as versioning and en- vironments. It also offers a concept which is easy to use and understand and, therefore, enables fast prototyping. Another important aspect working in Dialogflow's favor is their free Standard Edition, which comes with a sufficient quota of text requests. 24 https://dialogflow.com/, visited on 2019-03-31 iJIM ‒ Vol. 14, No. 18, 2020 25 https://dialogflow.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Dialogflow's Messenger and webchat integrations, however, provide only little room for customization and lack support for advanced features such as message receipts (Fa- cebook Messenger) and UI templates (webchat). Therefore, the chatbot uses a dedicated backend to handle incoming messages. On the user side, a client application (webchat or Facebook Messenger) is used. In the case of the webchat, communication happens directly between the server and the client. When using Facebook Messenger, client requests are first sent to Facebook's server before reaching Theodore's backend and vice versa. Fig. 3. Communication structure between a user and the chatbot on a webchat. Figure 3 shows the chronological structure of the communication between a user and the chatbot on the webchat. First, the client sends the user's input to the backend. The backend then sends the request to a NLU service, which extracts an intent and possible parameter to get to know what the user wants. If necessary, additional APIs are visited. For example, the latest blog posts are retrieved from the website using the Wordpress REST API. Subsequently, a proper response is created. After all this processing, the response is sent back to the client and presented to the user. The chatbot has to access different APIs in order to provide all required features. Wordpress's REST API is used to retrieve blog posts as well as open positions from the website. Mailchimp's REST API is used to subscribe users to the newsletter, if they want to. Other content and responses such as product and service descriptions are stored in files or in Dialogflow's intent handling. 26 http://www.i-jim.org Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Theodore is written from scratch using Node.js25 (version 8), Socket.IO26 and Di- alogflow. The backend handles connections from and to both Facebook Messenger and a custom webchat, written in HTML and TypeScript with the popular frontend fram- work Angular27. Communication is done with normal HTTP requests and websockets. The chatbot was built with modularity in mind, in order to easily adapt to different needs and provide an extendable base for other projects. Adapters for other bot plat- forms, NLU services and analytics like Chatbase28 as well as API connectors and addi- tional features can be added via extensions. Currently, Theodore features scripts for Facebook Messenger, a webchat via Socket.IO and a complete adapter for Dialogflow. 4 Case Study In order to see how the chatbot performs and whether users accept it as an alternative to the website of CodeFlügel29, a user study was conducted. Tests took place from No- vember 6th to December 22nd, 2018 with individual participants. Users were chosen based on a specific user classification. The goal was to only introduce participants which are relevant to the company’s website (and chatbot respectively) for example, potential employees. Those users were asked to perform several common tasks on the company website and with a corresponding chatbot. Timings of those tasks and user feedback were recorded to get quantitative (speed) and qualitative (acceptance, dis- & advantages) results and see what tasks are better suited for the chatbot compared to the website. 4.1 Setup User selection: Participants were chosen to reflect the user base of the company's website. The relevant user groups have been explained in 3.1 Target Audience. A total of twenty (20) users were selected based on the previously outlined classifi- cation. This approach makes it possible to better adapt the selection of test subjects to the actual users of the website and to create more appropriate tasks. At least five par- ticipants per user type were among them. The age ranged from 20 to 33, while the average user was 27.7 years old. 60% of the users had a bachelor or master's degree while another 25% were still working on their bachelor (with 20% working on their master's degree). 70% had a technical background (mostly IT), either through work or education. 25 https://nodejs.org/, visited on 2019-03-31 26 https://socket.io/, visited on 2019-03-31 27 https://angular.io/, visited on 2019-04-04 28 https://chatbase.com/, visited on 2019-03-31 29 https://www.codefluegel.com, visited on 2018-10-28 iJIM ‒ Vol. 14, No. 18, 2020 27 https://nodejs.org/ https://socket.io/ https://angular.io/ https://chatbase.com/ https://www.codefluegel.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Preparation: At the beginning of the test session, users were greeted and given a summary about the project as well as about the user study and the upcoming steps. Then they were asked to fill out a consent form and a questionnaire about their background, knowledge in different technologies such as chat applications, augmented reality and programming as well as their expectations of a company's website and chatbot. The goal was to collect enough background information about the users to be able to find possible connections between the use of the chatbot or the website and the different backgrounds. Setup and tasks: The test setup consisted of a workspace with a computer, two monitors, a keyboard and a mouse. The participants were placed in front of the com- puter while the facilitator was sitting to their left, a little further back, to observe the test person's actions and measure the time of each task. The right monitor showed in- structions for the tasks, while the left one was used to perform the tasks. The actions on the left monitor were captured with the screen capturing software OBS Studio30. Table 2. This table shows the tasks users have to complete with both the website and the chatbot. Task # Description and Goal 1 Task: Find out what CodeFlügel does or which services they provide. Goal: “Developer of Mobile Apps, Web and Augmented/Virtual Reality.” 2 Task: Find out where CodeFlügel’s office is and what phone number you can call. Goal: User finds address and phone number. 3 Task: Find out which companies CodeFlügel has already implemented projects for. Goal: User finds reference page/list. 4 Task: Find and open the latest blog entry. Goal: User opens the latest blog entry. 5 Task: Sign up for the newsletter with the e-mail address .@code fluegel.com. Goal: User signs up for the newsletter. 6 Task: Find out if and which jobs are currently available. Goal: User finds job page/list. 7 Task: Find a way to try Augmented Reality (AR) for yourself. Goal: User finds demo app. 8 Task: Find at least one Augmented Reality (AR) project created by CodeFlügel. Goal: User names at least one (1) Augmented Reality (AR) project. 9 Task: Find out what Augmented Reality (AR) actually is. Goal: User opens AR explanation. A list of nine typical tasks was defined (see Table 2), taking into account the various user groups and the target audience of the company's website. The list included things such as looking for the latest blog post, searching for job openings and finding at least one completed augmented reality project of the company. First, the tasks had to be per- formed on the company's website. In order to increase the quality of feedback and ob- servations, users were asked to verbalize their thoughts, similar to so called thinking- aloud-tests. 30 https://obsproject.com, visited on 2019-01-04 28 http://www.i-jim.org https://obsproject.com/ Paper—Chatbots for Brand Representation in Comparison with Traditional Websites Each task instruction had to be read out loud by the participants. After that, a timer was started by the facilitator. The user had to clearly state the solution to the given task and, if it was the right one, the timer was stopped. When all nine tasks were completed, the same tasks had to be performed using the chatbot instead of the website. Feedback and interviews: After the completion of the tasks, another questionnaire had to be filled out. The questionnaire was there to gather the participant's feedback, recognize potential flaws in the chatbot design and measure the acceptability of the company chatbot. An interview was then conducted to clear up any potential ambigui- ties and obtain better-quality feedback. 4.2 Results When asked what users expect from a chatbot and website of a company, the results were pretty close. Information about Products & Services and Contact Details were expected by all users from the website, while 70% (Contact Details) to 85% (Products & Services) expected the same from the chatbot. References, Jobs and Blog & Articles only amounted to 20% – 55% of the users' expectations. In terms of speed of the task completion, 80% of participants were faster using the chatbot. The average difference in time needed for all tasks was 2 minutes and 54 sec- onds in favor of the chatbot. The average completion time of all tasks was 3m 38s with the chatbot and 6m 33s with the website, while the averages for one task were 24 sec- onds (chatbot) and 43 seconds (website). Fig. 4. Average completion time per task for both platforms Six out of the nine tasks were completed faster with the chatbot, although two of those six were only completed about six seconds faster. Tasks involving the website iJIM ‒ Vol. 14, No. 18, 2020 29 Paper—Chatbots for Brand Representation in Comparison with Traditional Websites were faster or equally fast when a corresponding and self-explanatory (e.g. Blog) menu entry existed, while tasks seeking specific information not immediately accessible by the main menu (tasks 1, 7, 8 and 9) where significantly faster achieved with the chatbot. This is similar to what [10] concluded in their research. In terms of perceived completion time, 80% of test users thought they were faster using the chatbot. 75% correctly assessed the chatbot as being faster, while one of the participants thought completion time was faster, but the opposite was true. Another one thought their website tasks were completed faster, but in reality, the chatbot run was the quicker one. 15% of the test users believed to have been equally quick with both platform tasks while actually being faster with the website. Although the chatbot matched every participant's expectations, seven out of the 20 users said the input-to-intent matching could be improved, as the chatbot did not un- derstand every message and reformulating the request was necessary. 80% of the users found the chatbot more entertaining than the website vice versa, only 10% were more entertained using the website. 40% deemed the chatbot more modern, while none said anything similar about the website. The chatbot was also described as fast by 50%, while only 10% used this adjective for the website. 60% of the participants said they preferred a chatbot with informal communication. The other 40% did not care about whether the chatbot approached the conversation formally or informally. This pretty much confirms what [16] published in their study. [17] showed that humans tend to use shorter messages when conversing with chat- bots. Our study revealed that 50% of users relied on a combination of whole sentences and simple keywords to communicate with the chatbot. 30% used only sentences, while 20% based their messages solely on keywords. Half the users thought the website and chatbot were equally appealing to them. The website was favored by 25% of the participants, partially due to being used to websites or due to being able to explore the content more freely. [10] suspected similar reasons for favoring websites over chatbots in their study. The other 25% found the chatbot more appealing. Reasons to favor the chatbot were instantaneous answers to specific questions without the hassle of clicking through menus and searching through pages, as well as its interactive nature, which resulted in more fun for some of the participants. Overall, the company chatbot was received positively and 85% of participants could see themselves using more chatbots in the future. Despite extensive background checks of the participants, no correlation between the time needed to complete the tasks and the level of IT knowledge was observed. 5 Conclusion In order to compare the usage of chatbots to traditional websites, a specific company chatbot was created and a case study was conducted. Typical use cases of a company chatbot were worked out in order to compare such a chatbot with the existing company website and measure its performance. The user study showed that chatbots are accepted by users. The users provided pos- itive feedback about the chatbot, only some had minor issues with the matching of their 30 http://www.i-jim.org Paper—Chatbots for Brand Representation in Comparison with Traditional Websites queries and the size of the chat window. However, this could be improved easily by slightly changing the web chat design and providing more training data to the chatbot and the respective NLU engine. The study also revealed that it is important to train for whole sentences as well as key words, as both are commonly used by users. This was also discovered by [17]. The chatbot was faster than the website, especially when it comes to finding specific information about the company, and the users perceived it as more entertaining. Thus, it could potentially help companies in reaching new or younger audiences and in improving specific parts of their websites, such as the FAQs. 6 Future Perspectives Building on this study, further testing of the impact of UI improvements can be done. Natural Language Processing and matching of user inputs to intents also bear potential to increase the acceptance and enhance the user experience. In general, more chatbots could be tested and more users from different backgrounds and age groups could pro- vide more valuable insight. Another aspect to investigate further is the effect of different chatbot personas on the user experience and the influence of brand perception. One could also evaluate the pos- sibilities of distributing and marketing bots. Do chatbot "marketplaces" provide a sim- ilar exposure to mobile app stores such as Google Play and Apple App Store? Conversion rates and other Key Performance Indicators (KPIs) are things that could be explored in more detail as well. These analytics could show whether chatbots are economically feasible and here to stay. 7 References [1] USA Today (Mar. 30, 2016). Microsoft CEO Nadella: “Bots are the new apps”. Ed. By Theresa Chong. USA Today. [Online], Available: https://eu.usatoday.com/story/tech/news/ 2016/03/30/microsof-ceo-nadella-bots-new-apps/82431672/ [Accessed Jan. 28, 2019] [2] Comscore (Sept. 22, 2015). The 2015 U.S. Mobile App Report. Tech. rep. Comscore. [Online], Available: https://www.comscore.com/Insights/Presentations-and-Whitepapers/ 2015/The-2015-US-Mobile-App-Report [Accessed Jan. 24, 2019] [3] Klopfenstein, Lorenz Cuno et al. (June 2017). “The Rise of Bots: A Survey of Conversa- tional Interfaces, Patterns, and Paradigms.” In: Proceedings of the 2017 Conference on De- signing Interactive Systems. DIS ’17, pp. 555–565. ISBN: 978-1-4503-4922-2.doi: https://doi.org/10.1145/3064663.3064672 [4] Hootsuite and We Are Social (2019). Digital 2019. Global Digital Overview. Tech. rep. Mindbowser. [Online], Available: https://datareportal.com/reports/digital-2019-global-digi- tal-overview [Accessed Feb. 2, 2019] [5] Nadella, Satya (Mar. 25, 2016a). Build 2016. Keynote Presentation. Microsoft.[Online], Available: https://channel9.msdn.com/Events/Build/2016/KEY01#time=1h41m11s [Ac- cessed Feb. 13, 2019] [6] Chatbots Magazine and Anatoly Khorozov (Mar. 1, 2017). Trends Driving the Chatbot Growth. [Online], Available: https://chatbotsmagazine.com/trends-driving-the-chatbot- growth-77b78145bac [Accessed Mar 14, 2019] iJIM ‒ Vol. 14, No. 18, 2020 31 https://eu.usatoday.com/story/tech/news/2016/03/30/microsof-ceo-nadella-bots-new-apps/82431672/ https://eu.usatoday.com/story/tech/news/2016/03/30/microsof-ceo-nadella-bots-new-apps/82431672/ https://www.comscore.com/Insights/Presentations-and-Whitepapers/2015/The-2015-US-Mobile-App-Report https://www.comscore.com/Insights/Presentations-and-Whitepapers/2015/The-2015-US-Mobile-App-Report https://doi.org/10.1145/3064663.3064672 https://datareportal.com/reports/digital-2019-global-digital-overview https://datareportal.com/reports/digital-2019-global-digital-overview https://channel9.msdn.com/Events/Build/2016/KEY01#time=1h41m11s https://chatbotsmagazine.com/trends-driving-the-chatbot-growth-77b78145bac https://chatbotsmagazine.com/trends-driving-the-chatbot-growth-77b78145bac Paper—Chatbots for Brand Representation in Comparison with Traditional Websites [7] Shawar, Bayan Abu and Eric Atwell (Sept. 2015). “A chatbot as a Question Answering Tool.” In: 2015 International Conference on Advances in Software, Control and Mechanical Engineering. 2015 International Conference on Advances in Software, Control and Mechan- ical Engineering, pp. 1–6. ISBN: 978-93-84422-37-0. https://doi.org/10.17758/ur.u09 15120 [8] Xu, Anbang et al. (May 2017). “A New Chatbot for Customer Service on Social Media.” In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017 CHI Conference on Human Factors in Computing Systems, pp. 3506–3510. ISBN: 978-1- 4503-4655-9. https://doi.org/10.1145/3025453.3025496 [9] Puscher, Frank (2018). “Holpriger Start. Chatbots für Unternehmen.” In: iX 6/2018, pp. 58– 62 [10] Beriault-Poirier, Amélie, Sandrine Prom Tep, and Sylvain Sénécal (2019). “Putting Chatbots to the Test: Does the User Experience Score Higher with Chatbots Than Websites?” In: Human Systems Engineering and Design. Springer International Publishing, pp. 204–212. ISBN: 978-3-030-02053-8. https://doi.org/10.1007/978-3-030-02053-8_32 [11] Heckmann, Jörn and Michael Kraus (2018). “You, Robot. Rechtliche Risiken beim Einsatz von Chatbots.” In: iX 6/2018, pp. 70–72 [12] The Guardian (Mar. 24, 2016). “Tay, Microsoft’s AI chatbot, gets a crash course in racism from Twitter”. Ed. by Elle Hunt. The Guardian. [Online], Available: https://www.thegua rdian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-rac- ism-from-twitter. [Accessed Feb 22, 2019]. https://doi.org/10.1145/3008665.3008674 [13] Lebeuf, Carlene, Margaret-Anne Storey, and Alexey Zagalsky (Dec. 25, 2017). “Software Bots.” In: IEEE Software vol. 35 (1), pp. 18–23. https://doi.org/10.1109/ms.2017.4541027 [14] Araujo, Theo (Aug. 2018). “Living up to the chatbot hype: The influence of anthropo- morphic design cues and communicative agency framing on conversational agent and com- pany perceptions.” In: Computers in Human Behavior. Vol. 85, pp. 183–189 https://doi. org/10.1016/j.chb.2018.03.051 [15] Ciechanowski, Leon et al. (Mar. 2019). “In the shades of the uncanny valley: An experi- mental study of human–chatbot interaction.” In: Future Generation Computer Systems. Vol. 92, pp. 539–548. https://doi.org/10.1016/j.future.2018.01.055 [16] Survata and LivePerson (Apr. 2017). How Consumers View Bots In Customer Care. Tech. rep. LivePerson. [Online], Available: https://www.liveperson.com/resources/reports/bots- in-customer-care/ [Accessed Feb 20, 2019] [17] Hill, Jennifer, W. Randolph Ford, and Ingrid G. Farreras (2015). “Real conversations with artificial intelligence: A comparison between human–human online conversations and hu- man–chatbot conversations.” In: Computers in Human Behavior. Vol. 49, pp. 245–250. https://doi.org/10.1016/j.chb.2015.02.026 8 Authors Johannes Kühnel is currently working as a backend developer at CodeFluegel GmbH in Graz, Austria. He is mostly responsible for developing REST APIs and DevOps related work (e-mail: johannes@kuehnel.co.at). Markus Ebner is currently working as a Researcher in the Department Educational Technology at Graz University of Technology (e-mail: markus.ebner@tugraz.at). He deals with e-learning, mobile learning, technology enhanced learning and Open Educa- tional Resources. His focus is on Learning Analytics at K-12 level. In addition, several 32 http://www.i-jim.org https://doi.org/10.17758/ur.u0915120 https://doi.org/10.17758/ur.u0915120 https://doi.org/10.1145/3025453.3025496 https://doi.org/10.1007/978-3-030-02053-8_32 https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter https://www.theguardian.com/technology/2016/mar/24/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter https://doi.org/10.1145/3008665.3008674 https://doi.org/10.1109/ms.2017.4541027 https://doi.org/10.1016/j.chb.2018.03.051 https://doi.org/10.1016/j.chb.2018.03.051 https://doi.org/10.1016/j.future.2018.01.055 https://www.liveperson.com/resources/reports/bots-in-customer-care/ https://www.liveperson.com/resources/reports/bots-in-customer-care/ https://doi.org/10.1016/j.chb.2015.02.026 mailto:johannes@kuehnel.co.at mailto:markus.ebner@tugraz.at Paper—Chatbots for Brand Representation in Comparison with Traditional Websites publications in the area of Learning Analytics were published and workshops on the topic were held. Martin Ebner is with the Department Educational Technology at Graz University of Technology, Graz, Austria. (e-mail: martin.ebner@tugraz.at). As head of the Depart- ment, he is responsible for all university wide e-learning activities. He holds an Assoc. Prof. on media informatics and works at the Institute of Interactive Systems and Data Science as senior researcher. For publications as well as further research activities, please visit: http://martinebner.at. Article submitted 2020-01-26. Resubmitted 2020-07-24. Final acceptance 2020-07-29. Final version pub- lished as submitted by the authors. iJIM ‒ Vol. 14, No. 18, 2020 33 mailto:martin.ebner@tugraz.at http://martinebner.at/