AoIR2020-submission-final.pdf Selected Papers of #AoIR2020:  The 21 ​st ​ Annual Conference of the  Association of Internet Researchers Virtual Event / 27-31 October 2020 THE GREAT RANDOMIZER: USING VIRTUAL AGENTS FOR AUDITING THE EFFECTS OF YOUTUBE RECOMMENDATION ALGORITHM ON IDEOLOGICALLY-CHARGED NEWS CONTENT DISTRIBUTION Mykola Makhortykh University of Bern Aleksandra Urman University of Bern In recent years, algorithmic filtering and ranking of digital content on social media, in the news, and on streaming platforms, increasingly shape user information behaviour, even while users themselves are often are not aware of the influence of algorithms (i.e., Bodle, 2014; Eslami et al., 2015; Zanker et al., 2019). Given the rapid development of algorithmic distribution systems and the scope of influence they have on users’ information diets, and as a consequence, on their perception of important societal matters, it is necessary to research how these algorithms function, which effects they might have, and whether they are biased in certain ways (Loosen et al., 2016). The algorithms used by online platforms such as Facebook or YouTube rely on state-of-the-art big data and machine learning techniques. Because of their complexity and multiple variables involved in the process of automated curation of information distribution, these algorithms often turn into black boxes even for the developers who created them (Pasquale, 2015). In addition, these algorithms are the intellectual property of corporations which developed them, meaning that researchers do not have access to their source code and can only obtain limited data about the ways these algorithms function. For these reasons, it is difficult to study the effects of algorithmic filtering and ranking on the distribution of societally relevant information, but not impossible - researchers can use algorithmic auditing in order to infer the patterns in the ways the algorithms function (Mittelstadt, 2016). Suggested Citation (APA): Makhortykh, M., Urman, A. (2020, October). ​The great randomizer: using virtual agents for auditing the effects of Youtube recommendation algorithm on ideologically-charged news content distribution ​. Paper presented at AoIR 2020: The 21 ​th ​ Annual Conference of the Association of Internet Researchers. Virtual Event: AoIR. Retrieved from http://spir.aoir.org. In our paper, we introduce a novel approach for large-scale algorithmic auditing based on the use of virtual agents. In contrast to earlier studies into algorithmic ranking (i.e., Hannák et al., 2017; Kulshrestha et al., 2017; Puschmann, 2019), which tend to either rely on small data samples or do not control for external variables (i.e., the effect of time on the results), we developed infrastructure that allows conducting auditing experiments in a fully controlled testing environment. Specifically, our infrastructure enables simulating the synchronous activity of a large number of autonomous agents and tracking their browsing behaviour as well as interactions with online platforms (Makhortykh, Urman, & Ulloa, 2020). As a case study, we deploy the larger sample of virtual agents (n=200) to investigate how YouTube recommendation algorithm filters and ranks information related to the far-right terrorist attack in the German city of Halle in 2019. Using a series of experiments in the controlled environment, we investigate how content recommendations produced by YouTube in relation to the attack differ depending on the kinds of videos about Halle watched by the agents. Specifically, we compare differences between recommendations for news videos produced by mainstream news agencies (e.g. ZDF) and political bloggers with different ideological positions. Our findings highlight several important aspects of YouTube recommendation algorithms in relation to terrorism-related news distribution. Firstly, we identify the high degree of randomization that leads to fundamentally different recommendation trajectories despite the same starting conditions (i.e. the uniform pre-history of watched content) and identical behaviour patterns. Secondly, we find significant differences in the algorithmic filtering/ranking between the browsers, in particular higher consistency of recommendations for the agents who interacted with YouTube via Firefox. Finally, we find that the recommendations for the agents watching right-leaning news videos on the Halle attack are slightly more consistent than those for the mainstream and left-leaning ones. The latter observation suggests that the degree of randomization for recommendations related to right-wing news content on YouTube is lower, hence increasing the probability of formation of an isolated information environment or a “filter bubble” (Pariser, 2011). However, considering that even in this case YouTube recommendations tend to be strongly randomized and predominantly entertainment- and not politics-related, our findings suggest that YouTube algorithm is more of a great randomizer than a “great radicalizer” (Tufekci, 2018). References Bodle, R. (2014, November 13). ​Predictive Algorithms and Personalization Services on Social Network Sites: Implications for Users and Society​. The Ubiquitous Internet. https://www.taylorfrancis.com/ Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., & Sandvig, C. (2015). “I always assumed that I wasn’t really that close to [her]”: Reasoning about Invisible Algorithms in News Feeds. ​Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15​, 153–162. https://doi.org/10.1145/2702123.2702556 Hannák, A., Sapieżyński, P., Khaki, A. M., Lazer, D., Mislove, A., & Wilson, C. (2017). Measuring Personalization of Web Search. ​ArXiv:1706.05011 [Cs]​. http://arxiv.org/abs/1706.05011 Kulshrestha, J., Eslami, M., Messias, J., Zafar, M. B., Ghosh, S., Gummadi, K. P., & Karahalios, K. (2017). Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. ​Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing​, 417–432. https://doi.org/10.1145/2998181.2998321 Loosen, W., Bastos, M. T., Puschmann, C., Hasebrink, U., Hölig, S., Merten, L., Schmidt, J.--H., Kinder​-Kurlanda, K. E., & Weller, K. (2016). Caught in a feedback loop? Algorithmic personalization and digital traces. ​AoIR Selected Papers of Internet Research​.​ ​https://journals.uic.edu/ojs/index.php/spir/article/view/8491 Makhortykh, M., Urman, A., & Roberto, U. (2020). How search engines disseminate information about COVID-19 and why they should do better. ​The Harvard Kennedy School (HKS) Misinformation Review​. https://misinforeview.hks.harvard.edu/article/how-search-engines-disseminate-informati on-about-covid-19-and-why-they-should-do-better/ Mittelstadt, B. (2016). Automation, Algorithms, and Politics| Auditing for Transparency in Content Personalization Systems. ​International Journal of Communication​, ​10 ​(0), 12. Puschmann, C. (2019). Beyond the Bubble: Assessing the Diversity of Political Search Results. ​Digital Journalism​, ​7 ​(6), 824–843. https://doi.org/10.1080/21670811.2018.1539626 Pariser, E. (2011). ​The Filter Bubble: What The Internet Is Hiding From You​. Penguin UK. Pasquale, F. (2015). ​The Black box society: The secret algorithms that control money and information​ (First Harvard University Press paperback edition). Harvard University Press. Tufekci, Z. (2018, March 10). Opinion | YouTube, the Great Radicalizer. ​The New York Times​. https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-radical.html Zanker, M., Rook, L., & Jannach, D. (2019). Measuring the impact of online personalisation: Past, present and future. ​International Journal of Human-Computer Studies​, ​131 ​, 160–168. ​ ​https://doi.org/10.1016/j.ijhcs.2019.06.006 n.