Journal of Applied Engineering and Technological Science Vol 4(1) 2022: 539-546 539 MEASUREMENT AND ANALYSIS OF TOURISM WEBSITE USER EXPERIENCE USING USABILITY TECHNIQUES Rika Perdana Sari1*, Silvana Rasio Henim2 Politeknik Caltex Riau rika@pcr.ac.id Received : 20 November 2022, Revised: 14 December 2022, Accepted : 14 December 2022 *Corresponding Author ABSTRACT A tourist destination is a geographical area within one or more administrative areas in which there are tourist attractions, public facilities, tourism facilities, accessibility, and communities that are interrelated and complement the realization of tourism. A website is a form of communication media that can provide certain information, which can be accessed by many people anytime and anywhere. The Regional Government (pemda) of Pekanbaru City has provided tourism information services in the form of tourist destinations and tourism activities on the website. Based on the results of preliminary observations made on the tourism website of the Pekanbaru City government, it was found that the tourism information provided was not up to date, and there were still many tourist objects in Pekanbaru that had not been published, so the information delivery process was not optimal and ineffective. The effectiveness of using the website must pay attention to usability of its use. Usability is a quality level of a system that is easy to learn, easy to use, and encourages users to use the system as a tool in completing tasks. Usability can be measured based on the quality of the user experience when interacting with the system operated by the user himself. Usability measurement on this tourism website is carried out using the System Usability Scale (SUS) measurement tool. Based on the measurement results using SUS, a score of 65.87 was obtained, where 10 of the statement items were declared valid and reliable. The score obtained shows that in terms of user satisfaction, it is stated to be included in the Marginal High category, predicate D with the adjective range Ok. This shows that the user's subjective assessment illustrates that the Pekanbaru City government's tourism website is less effective, efficient, and satisfying for users. Keywords: website, tourism, usability, system usability scale 1. Introduction The definition related to tourism contained in the tourism and creative economy Ministerial Regulation No. 9 of 2021 concerning guidelines for sustainable tourism destinations is intended to be a variety of tourism activities, which are supported by various facilities and services provided by the community, entrepreneurs, Government and Regional Governments (MENPAREKRAF, 2021). Tourists, whether local, national, or international, are people who carry out tours in the context of carrying out travel activities, whether carried out individually or in groups, by visiting certain places, for recreational purposes, personal development, or studying the unique tourist attractions visited in a temporary period (PEMKO PEKANBARU, 2021) (Pratama, Misnawati, & Renaldi, 2022). Tourists usually make visits to tourism destinations, hereinafter referred to as tourism destinations. Tourism Destinations are geographical areas that are in one or more administrative areas in which there are tourist attractions, public facilities, tourism facilities, accessibility, and communities that are interrelated and complement the realization of tourism. In addition to natural conditions that have a special attraction for tourists to visit, tourism destinations can also be formed based on facilities created by humans, art and culture, and national history (Addin Maulana, 2022) (Ilham Junaid, 2022). Nowadays, where the use of information technology is an inseparable part of daily human life, a tourist will definitely use the internet to access various social media or regional tourism websites to find tourist destinations in an area (Filemon Duwitau, 2020) (Sagita Utarki, 2020) (Sandra J Kuryanti, 2018). Research conducted by Sri Wulandari et al. states that the use of the website has proven to be effective in promoting and disseminating information. Researchers measured the effectiveness of using digital media as a means of university promotion and information. Based on measurements taken on 120 respondents, it was found that from several Sari et al … Vol 4(1) 2022 : 539-546 540 social media used for promotion, website media showed very significant results. This means that new students at Gunung Jati University get campus information through the website, and the website has proven to be effective for use as a media for promotion and information dissemination (Wulandari, Heryani, & Nurlia, 2021). The effectiveness of using the website was also proven by Mardian Herri Pratama et al. for promotional media in the East Surabaya Village, Kec. Appeal to the Supreme Court of OKU Selatan Prov. South Sumatra. The results of the study show that social media is quite effective in marketing a product and can be utilized for business needs, but account management must be optimized (Pratama, Misnawati, & Renaldi, 2022). The website is used as a form of communication media that can provide certain information, which can be accessed by many people anytime and from anywhere. Based on the good practices of the website as a communication medium, the effectiveness of its use must also pay attention to the usability of its utilization. Usability is a quality level of a system that is easy to learn, easy to use, and encourages users to use the system as a positive tool in completing tasks. Usability can also be interpreted as a measure in which users can access the functionality of a system effectively, efficiently, and satisfactorily in achieving certain goals. The Regional Government of Pekanbaru City has provided tourism information services in the form of a website which can be accessed on the https://pariwisata.pekanbaru.go.id/ home page. This tourism website is intended as a medium for delivering information on tourist destinations and tourism activities. Based on the results of preliminary observations made on this tourism website, it was found that information about tourist destinations and tourism activities was not up to date, and there were still many tourist objects in Pekanbaru that had not been published so the information delivery process was not optimal and ineffective in presenting the need for information by prospective tourists who will visit Pekanbaru. However, usability can be measured based on the quality of the user experience when interacting with the system operated by the user himself. User experience, in this case, tourists and their perceived interaction with the website, the measurement process is then carried out using the System Usability Scale (SUS) measuring instrument. SUS is proven to be valid and reliable to be used to measure the usability of a website (Sari & Henim, 2021). 2. Literature Review Usability Testing Usability is a quality that examines and measures the ease of display used by users. Usability refers to the user experience when interacting with a product or system, including websites, software, devices or applications (Supriyatna, 2018) (Hendra & Arifin, 2018). Usability focuses on creating systems that are easy to learn and use (Soejono, Setyanto, & Sofyan, 2018). The following are usability indicators according to Nielsen: 1. Efficiency measures the speed in completing certain tasks after studying the system 2. Satisfaction measures how satisfied users are in using th (Addin Maulana, 2022)e system 3. Memorability measures whether the application can be recalled by the user 4. Errors measure how many user errors in using the system 5. Learnability measures the user's level of ease in completing tasks that must be done (Supriyatna, 2018) (Kesuma, 2021). Usability can be measured in 2 ways, that are using usability metrics and using their own assumptions. Usability metrics are used to help reach decisions. Usability provides answers to questions such as, do users like the product? is this product more efficient to use than similar products? can the usability of this product be compared to competition? and so on (Hadi, Az- Zahra, & Fanani, 2018). System Usability Scale (SUS) The System Usability Scale (SUS) is a usability testing tool that is reliable, popular, effective, and inexpensive. SUS developed by John Brooke in 1986. SUS has 10 question instruments as shown in the following table: https://pariwisata.pekanbaru.go.id/ Sari et al … Vol 4(1) 2022 : 539-546 541 Table 1 - SUS Question Instruments No Statement List 1 I will often use/visit this system 2 I think the system is too complex (contains a lot of unnecessary things) 3 I think the system is easy to navigate 4 I need technical assistance to use/explore the system 5 I think the functions/features provided on the system are well designed and prepared 6 I think there are too many inconsistencies on the system 7 I feel that most people will find it easy to use/explore the system quickly 8 I think the system is very complicated to navigate 9 I feel very confident exploring the system 10 I need to learn many things before I can properly explore system The results of calculations using SUS will be converted into a value, which can be used as a consideration to determine whether an application is feasible or not feasible to implement (Pudjoatmodjo & Wijaya, 2016) (Sidik, 2018). SUS consists of 5 scales, i.e 1 to 5. A value of 1 means strongly disagree and a value of 5 means strongly agree as shown in the following Figure 1 (Saputra, 2019). Fig. 1. SUS Scale 3. Research Methods Approach and Research Object The research conducted is quantitative research, using survey techniques. Survey research was conducted using a questionnaire tool to collect data from a predetermined population sample. The questionnaire is intended to obtain information from a number of respondents who represent a certain population, in this case, a group of respondents who have carried out tourism activities. The object of this research is the tourism website of the Pekanbaru city government, which includes all interfaces from the existing display. Research Stages In this research, several stages were carried out, starting with identifying the problem, followed by conducting a literature study related to usability testing and also a study of the observed object, in this case, the Pekanbaru city government's tourism website. An overview of this research methodology can be shown in Figure 2. Fig. 2. Research Methodology Sari et al … Vol 4(1) 2022 : 539-546 542 Research Instruments At the usability testing stage, a number of tasks are given to website users. The tasks given at the usability stage are shown in Table 2. Table 2 - List of Usability Testing Tasks No Task 1 The user opens the Pekanbaru city government tourism website 2 The user selects an information menu related to a tourist attraction 3 The user selects a tourist attraction and reads the detailed information contained on the page of the selected tourist attraction 4 Users search for and read information on accommodation 5 The user selects the travel and travel menu 6 Users choose a travel and travel agent, and read the detailed information contained on the page that is displayed 7 Users select the event menu and search for events based on the year they were held 4. Results and Discussions After the respondent has carried out all the tasks, the process of measuring the usability of the respondent is carried out. Measurement of respondents' perceptions was carried out using the System Usability Scale (SUS). SUS was developed by John Brooke in 1986 and used to measure system usability from the user's perspective. SUS consists of 10 statements (Brooke,1996) (Bangor, Kortum, & Miller, 2009)(Brooke,2013) , as shown in Table 3. Table 3 - List of SUS Questions Code Statement List Q1 I will often use/visit the Pekanbaru City tourism website Q2 I think the Pekanbaru City tourism website is too complex (contains a lot of unnecessary things) Q3 I think the Pekanbaru City tourism website is easy to navigate Q4 I need technical assistance to use/explore the Pekanbaru City tourism website Q5 I think the functions/features provided on the Pekanbaru City tourism website are well designed and prepared Q6 I think there are too many inconsistencies on the Pekanbaru City tourism website Q7 I feel that most people will find it easy to use/explore the Pekanbaru City tourism website quickly Q8 I think the Pekanbaru City tourism website is very complicated to navigate Q9 I feel very confident exploring the Pekanbaru City tourism website Q10 I need to learn many things before I can properly explore Pekanbaru City tourism websites The following are the rules that need to be considered to calculate the SUS score: 1) Each question has an odd number, and the final score is the result of subtracting the user's score (x) minus 1. 2) Each question has an even number, and the final score is obtained from the subtraction of 5 minus the user's score (x). 3) The weighting of the SUS score is obtained from the sum of the user's overall scores multiplied by 2.5. 4) To calculate the SUS score, add up the score contributions of each item. (Ramadhan, Soedijono, & Pramono, 2019) (Wahyuningrum, Kartiko, & Wardhana, 2020) Sari et al … Vol 4(1) 2022 : 539-546 543 The results of the SUS calculation are then converted into ratings and letter grades. The rating indicates the usability level in the form of a percentage (%), while the letter grade indicates the usability level from class A to F. SUS weighting in the form of letter grades can be grouped into 5, namely A, B, C, D, and F, as shown in Figure 3. Fig. 3. SUS Score ( (Bangor, Kortum, & Miller, 2009) (Sauro, 2011) The discussion of the research results obtained can be presented in the form of a theoretical description, both qualitatively and quantitatively. In practice, this section can be used to compare the results of the research obtained in the current research with the results of the research reported by previous researchers referred to in this study. Scientifically, the results of research obtained in the study may be new findings or improvements, affirmations, or rejection of a scientific phenomenon from previous researchers. Usability Testing: System Usability Scale Tests were conducted on 166 respondents, who had the criteria of having traveled out of town on holidays and accessed tourism websites to obtain information related to tourist attractions. Based on the processing carried out on the answers given by the respondents using the SUS measurement tool, the results are shown in Table 4. Table 4 - Recapitulation of the results of calculating the SUS score Respondent SUS Calculation Result Score SUS Score Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 R1 3 3 3 4 4 4 3 4 4 3 87.50 R2 3 3 3 4 3 3 3 4 2 3 77.50 R3 3 3 3 3 3 3 3 3 3 1 70.00 R4 3 2 2 1 3 1 4 1 4 0 52.50 R5 2 4 4 0 4 4 4 4 3 0 72.50 R6 3 2 2 2 3 2 3 2 2 1 55.00 R7 3 3 3 3 2 2 2 2 3 1 60.00 R8 3 2 2 3 3 3 2 3 1 1 57.50 R9 4 3 3 3 3 2 3 3 3 1 70.00 R10 3 3 3 4 4 3 3 4 3 2 80.00 … .. .. .. .. .. .. .. .. .. .. .. R166 4 0 4 0 4 0 4 0 4 0 50.00 Average SUS score 65.87 * Source: research data Validity Test A validity test is used to show the indicators measured are in accordance with the indicators to be studied. The measurement results are said to be valid if the correlation coefficient is at a significant level (0.05) to the total score. In this study, Pearson (2 tails) was used for the validation test. If the conditions Rcount > Rtable, then the results of the validity test were considered valid. Based on data from the ten statements in the questionnaire, it was found that the Rtable value was smaller than the Rcount value, so it could be stated that the questionnaire statements were considered valid. Sari et al … Vol 4(1) 2022 : 539-546 544 Table 5 - Results of the questionnaire data validity test Statement Code Rcount Rtable Description Q1 0.55 0.17 Valid Q2 0.70 0.17 Valid Q3 0.46 0.17 Valid Q4 0.90 0.17 Valid Q5 0.30 0.17 Valid Q6 0.76 0.17 Valid Q7 0.42 0.17 Valid Q8 0.58 0.17 Valid Q9 0.27 0.17 Valid Q10 0.82 0.17 Valid * Source: research data Reliability Test Reliability is closely related to consistency and level of accuracy. Measurements with high- reliability values mean showing consistency. A relatively high level of reliability is shown by the rxx value close to 1. In reliability testing, it can be said to be satisfactory if the Cronbach Alpha value is≥ 0.700. In this study, a reliability value of 0.71 was obtained for ten questionnaire statements, and this means that the results of the reliability test of this study can be said to be reliable. Table 6 - Questionnaire Data Reliability Test Results (Reliability Statistics) Method (a) Number of Items Description 0.637 10 Reliable * Source: research data SUS Outcome Analysis Measurement of the experience of users of the Pekanbaru city tourism website using the SUS measurement tool obtained a score of 65.87, where 10 of the statement items were declared valid and reliable from the process of distributing questionnaires to respondents. Besides that, the results obtained were analyzed by analyzing the SUS score from the perspective of acceptability, grade scale, and adjective rating, as shown in Figure 4. Fig. 4. SUS measurement of the Pekanbaru city tourism website from 3 perspectives (grade, adjective and acceptability) The Pekanbaru city SUS website score, as shown in Figure 3, gets a value of 65.87, which is included in the High Margin category. From the SUS Score, it is considered ok from an Adjective point of view. Meanwhile, based on the grade scale, as studied by Sauro in 2011, it can be categorized into five predicates, namely predicate A, predicate B, which is worth more than equal to (≥) 74 and less than equal to (≤) 80.3, predicate C, namely score has a range of more than (>) 68 and less than (<) 74, and predicate D, namely scores have a range of more than equal to (≥) 51 and less than (<) 68. Based on the results of the SUS website score getting predicate D, this can be interpreted from the perspective of the Pekanbaru City government tourism website Sari et al … Vol 4(1) 2022 : 539-546 545 user is considered less effective, efficient, and satisfying for users. In the future, it needs to be developed further so as to get better results, and further problems can be identified using other evaluation methods. 5. Conclusion Based on the research that has been done, it can be concluded that measuring the usability of the Pekanbaru city tourism website can be the first step in evaluating the website. The SUS score is 65.87, indicating that the website cannot be used properly, this can trigger a decrease in the number of users accessing the website. This can be shown from the processing of the fourth question, in which respondents stated that they really needed technical assistance to use/explore the Pekanbaru City tourism website. Based on the user satisfaction side, the results obtained are Margin High, predicate D, with the adjective range Ok. The increase user satisfaction, it is necessary to carry out further evaluation and development. The results of this study can be used as a basis for further research development processes in order to identify problems that cause user dissatisfaction. Therefore it is necessary to apply other evaluation methods to identify problems based on user needs, in this case, tourists visiting Pekanbaru. Acknowledgement - optional We thank the Research and Community Service Agency (BP2M) of the Caltex Riau Polytechnic for supporting the publication of research results that have been conducted by researchers. References Addin Maulana, D. O. (2022). IMPLIKASI KEBIJAKAN ATASTERBITNYA TRAVEL & TOURISM DEVELOPMENT INDEX2021 TERHADAP UPAYA PENINGKATAN DAYA SAING KEPARIWISATAANINDONESIA DI PASARGLOBAL. Jurnal Kepariwisataan Indonesia Vol 16 No 2, 149-162. Bangor, A., Kortum, P., & Miller, J. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123. Brooke, J. (1996). SUS-a quick and dirty usability scale. Usability evaluation in industry 189(194), 4-7. Brooke, J. (2013). SUS: a retrospective. Journal of Usability Studies, 8(2), 29-40. Filemon Duwitau, R. W. (2020). Sistem Informasi Pariwisata Daerah Kabupaten Nabire. INFORMATIKA DAN RPL, Vol. 2, No. 2, 104-112. Hadi, K., Az-Zahra, H., & Fanani, L. (2018, Feb). Analisis Dan Perbaikan Usability Aplikasi Mobile KAI Access Dengan Metode Usability Testing Dan Use Questionnaire. Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, 2(9), 2742-2750. Hendra, & Arifin, Y. (2018). Web-based Usability Measurement for Student Grading Information System. Procedia Computer Science, 135, 238-247. Ilham Junaid, M. D. (2022). MODEL PENGEMBANGAN INTERPRETASI PARIWISATA EDUKASI DI MUSEUM KOTA MAKASSAR. Jurnal Kepariwisataan Indonesia Vol 16 No 2, 216-236. Kesuma, D. (2021). Using System Usability Scale Method To Measure Usability Aspects On Online Learning Media at XYZ University. JATISI (Jurnal Teknik Informatika dan Sistem Informasi), 8(3), 1615-1626. MENPAREKRAF. (2021). Peraturan Menteri Pariwisata dan Ekonomi Kreatif/ Kepala Badan Pariwisata dan Ekonomi Kreatif Republik Indonesia, Nomor 9 Tahun 2021 Tentang Pedoman Destinasi Pariwisata Berkelanjutan. PEMKO PEKANBARU. (2021). Peraturan Daerah Kota Pekanbaru Nomor 1 Tahun 2021 Tentang Rencana Induk Pembangunan Kepariwisataan Daerah Kota Pekanbaru Tahun 2021-2036. Pratama, M., Misnawati, D., & Renaldi, M. (2022). Efektivitas Sosial Media Sebagai Media Promosi Desa Surabaya Timur. Jurnal Ilmu Sosial dan Humaniora, 1(1), 86-90. Pudjoatmodjo, B., & Wijaya, R. (2016). TES KEGUNAAN (USABILTY TESTING) PADA APLIKASI KEPEGAWAIAN DENGAN MENGGUNAKAN SYSTEM USABILTY Sari et al … Vol 4(1) 2022 : 539-546 546 SCALE (STUDI KASUS : DINAS PERTANIAN KABUPATEN BANDUNG). SEMNASTEKNOMEDIA. 4, pp. 37-42. Yogyakarta: Amikom Yogyakarta. Ramadhan, D. W., Soedijono, B., & Pramono, E. (2019, Desember). PENGUJIAN USABILITY WEBSITE TIME EXCELINDO MENGGUNAKAN SYSTEM USABILITY SCALE (SUS) (STUDI KASUS: WEBSITE TIME EXCELINDO). JIPI (Jurnal Ilmiah Penelitian dan Pembelajaran Informatika), 04(02), 139-147. Sagita Utarki, E. A. (2020). Sistem Informasi Pariwisata Berbasis Website Pada Taman Nasional Gunung Ciremai Jawa Barat. IJSE – Indonesian Journal on Software Engineering, Vol.6, No. 1, 19-32. Sandra J Kuryanti, N. I. (2018). Pembuatan Website Sebagai Sarana Promosi Pariwisata (Studi Kasus :Pantai Jatimalang, Purworejo). Publikasi Jurnal & Penelitian Teknik Informatika Volume 2 Nomor 2, 37-46. Saputra, A. (2019). Penerapan Usability pada Aplikasi PENTAS Dengan Menggunakan Metode System Usability Scale (SUS). JTIM : Jurnal Teknologi Informasi Dan Multimedia, 1(3), 206-212. Sari, R., & Henim, S. (2021). The Application Of System Usability Scale Method To Measure The Usability Of Electronic Learning System (E-Learning) Of Politeknik Caltex Riau. ILKOM Jurnal Ilmiah, 13(3), 266-271. Sauro, J. (2011). Measuring usability with the system usability scale (SUS). Retrieved from URL: Http://Www. Measuringusability. Com/Sus Sidik, A. (2018). Penggunaan System Usability Scale (SUS) Sebagai Evaluasi Website Berita Mobile. Technologia, Vol 9, No.2, 83-88. Soejono, A. W., Setyanto, A., & Sofyan, A. (2018). Evaluasi Usability Website UNRIYO Menggunakan System Usability Scale (Studi Kasus: Website UNRIYO). JTI Respati (J- urnal T-eknologi I-nformasi Respati), 13(1), 29-37. Supriyatna, A. (2018). Penerapan Usability Testing untuk Pengukuran Tingkat Kebergunaan Web Media of Knowledge. Jurnal Ilmiah Teknologi-Informasi dan Sains (TeknoIS), 8(1), 1-16. Wahyuningrum, T., Kartiko, C., & Wardhana, A. (2020). Exploring e-Commerce Usability by Heuristic Evaluation as a Compelement of System Usability Scale. 2020 International Conference on Advancement in Data Science, E-learning and Information Systems (ICADEIS) (pp. 1-5). Lombok: IEEE. Wulandari, S., Heryani, A., & Nurlia, T. (2021). Efektivitas Media Digital Dalam Meningkatkan Promosi Dan Informasi di Universitas Swadaya Gunung Jati Cirebon. Jurnal Ilmiah Publik, 9(2), 246-250.