Selected Papers of #AoIR2022: The 23rd Annual Conference of the Association of Internet Researchers Dublin, Ireland / 2-5 Nov 2022 Suggested Citation (APA): Burgess, J., Carah, N., Brown, M., Angus, D. (2022, November). #cottagecore, #futurecore, #sadcore: Using critical simulation to explore the interplay between machine vision and vernacular Instagram aesthetics. Paper presented at AoIR 2022: The 23rd Annual Conference of the Association of Internet Researchers. Dublin, Ireland: AoIR. Retrieved from http://spir.aoir.org. #COTTAGECORE, #FUTURECORE, #SADCORE: USING CRITICAL SIMULATION TO EXPLORE THE INTERPLAY BETWEEN MACHINE VISION AND VERNACULAR INSTAGRAM AESTHETICS Jean Burgess Queensland University of Technology Nicholas Carah University of Queensland Maria-Gemma Brown University of Queensland Daniel Angus Queensland University of Technology Introduction Instagram and parent company Meta use machine vision to recognise and label objects and features in images, for a range of purposes: generating automated text captions for accessibility; informing the recommendation engines that curate our feeds; and targeting advertising. Therefore, in using Instagram, we incidentally help train the platform’s machine vision systems; and these systems play a role in determining which images and ads we see. While haunted by the geneaologies and biases of its underlying training models (Denton et al., 2021), machine vision is increasingly used in more open-ended unsupervised modes that are oriented not toward detecting and recognising objects in images, but toward discovering patterns of similarity and association across sets of images. These patterns are used alongside other data signals to infer affinities and similarities among users via the content they interact with. Without necessarily fixing them as categories or giving them symbolic labels, platforms use these vectors of association to curate user feeds and target ads (Cotter et al., 2021). This work-in-progress paper is part of a multi-year project where we use a novel combination of computational and qualitative methods called ‘critical simulation’ (Burgess et al., 2021) to explore the interplay between everyday image-making practices and the algorithmic architecture of Instagram. At the heart of the project is a custom-built system that approximates the machine vision technologies in use by Instagram, informed by details gleaned from Meta’s own technical blogs and the broader technology press. Building on the first phase of the project, which explored the ability of machine vision to identify meaningful clusters of objects and environments given the Instagram images from a popular music festival (Carah, Angus & Burgess, 2022), this paper moves on to a harder problem: to understand the capacity of machine vision systems to recognise and reproduce the diverse vernacular aesthetics and affects (the ‘vibes’ and ‘moods’) associated with particular scenes on Instagram (Carah & Angus, 2018) - in this case, drawing on a case study of Instagram’s ‘-cores’ hashtags. The -cores case study On Instagram, ‘-cores’ hashtags proliferate around the the affects and aesthetics (the ‘vibes’) - associated with particular cultural scenes. They include #aestheticcore, #mallcore, #cybercore, #grungecore, #futurecore, #ravecore, #2000score, #nostalgiacore, #abandonedcore, #sanriocore, #scenecore, #sadcore and so on. There are hundreds of ‘-cores’ hashtags on the platform. ‘-cores’ are significant both because they continuously proliferate and because they express the commonplace sensibility of continuously shifting ‘vibes’ associated with each platform’s vernacular cultures of use (Davis, 2022; Gibbs et al., 2015). These collective image-making practices are also symptomatic and expressive of the deeper ‘structures of feeling’ (Coleman, 2018; Papacharissi, 2015) associated with rapidly emergent and competing currents of cultural change (Therieau, 2022). The most well-known recent example is #cottagecore, which began as a cosy, retro visual aesthetic for fashion and home decorating, crossed over into mainstream popular and celebrity culture via TikTok trends, memes, Spotify playlists, and Taylor Swift’s album Folklore, and in some spaces has come to represent an at-times counter-feminist white millenial ethos oriented around homemaking. For our purposes, the ‘-cores’ hashtags are particularly intriguing because while Instagram users engage in deliberate, coherent and patterned aesthetic practices, we also understand the ‘-cores’ as affective in the sense that they give expression to sensations that either cannot be fully captured and conveyed symbolically—as well as, in some cases, subcultural symbolic practices whose meanings are deliberately concealed from outsiders. The tendency of vernacular aesthetics to evade verbal description and symbolic representation may pose a significant challenge to machine vision systems, which are primarily trained to detect and match objects and faces. This leads us to the empirical question at the heart of this paper: how and to what extent are the various and overlapping aesthetics associated with the -cores salient to or reproducible by the machine vision technologies commonly used by platforms like Instagram? Investigation and analysis In order to address our empirical research questions, we first use our purpose-built machine vision system to undertake unsupervised clusterings of a sample of the 359,150 images associated with a curated set of 60 ‘-cores’ hashtags, which we have collected following a period of immersive qualitative investigation of the -cores phenomenon on the platform itself during 2021. Our system is not pre-trained on the images in our dataset, and nor does it have any information as to which captions, usernames or hashtags are associated with which images. We are therefore able to examine post-hoc whether the model, using only latent properties of the images themselves, clusters together images according to their hashtags or imposes entirely different logics of association. Figure 1: Our system provides an interactive interface to explore image clusters organised hierarchically according to the machine vision algorithm. On the left pane we see a cluster of >1000 images, organised into 16 sub clusters. Tool tips allow inspection of the mixture of hashtags that associate with images in any single cluster, and we can click into these high-level clusters to inspect further successive sub-clusters of images (as indicated via the right pane). We anticipate mixed results. The ‘-cores’ are partly characterised by discernible semiotic patterns that we might expect our model to cluster together. For instance, #cottagecore images feature visually distinctive soft lighting, cosy cardigans, warm tones, and botanicals. But, from the perspective of Instagram’s cultures of use, ‘-cores’ hashtags capture a fleeting aesthetic style or feeling that cannot easily be described on the basis of the distinctive objects or scenes depicted in the images, and therefore may remain inscrutable to our models, which remain biased toward objects and faces. We then undertake a close cultural analysis of the clusters the machine does produce, reading them through the lens of our existing knowledge of the -core hashtags. This enables us to speculate on how the platforms’ machine vision logics, once imposed on algorithmic feed curation, for example, might play a role in shaping Instagram’s platform aesthetics, and on internet culture more broadly. References Burgess, J., Angus, D., Carah, N., Andrejevic, M., Hawker, K., Lewis, K., … Li, L. (2021). Critical simulation as hybrid digital method for exploring the data operations and vernacular cultures of visual social media platforms. SocArXiv [preprint] https://doi.org/10.31235/osf.io/2cwsu Carah, N., & Angus, D. (2018). Algorithmic brand culture: participatory labour, machine learning and branding on social media. Media, Culture & Society, 40(2), 178-194. Carah, N., Angus, D., & Burgess, J. (2022). Tuning machines: an approach to exploring how Instagram’s machine vision operates on and through digital media’s participatory visual cultures. Cultural Studies https://doi.org/10.1080/09502386.2022.2042578 Coleman, R. (2018). Theorizing the present: digital media, pre-emergence and infra- structures of feeling. Cultural Studies 32(4): 600-622. Cotter, K., Medeiros, M., Pak, C., & Thorson, K. (2021). “Reach the right people”: The politics of “interests” in Facebook’s classification system for ad targeting. Big Data & Society 8(1), https://doi.org/10.1177/2053951721996046 Davis, A. (2022). A vibe shift is coming. Will any of us survive it? The Cut. Available online: https://www.thecut.com/2022/02/a-vibe-shift-is-coming.html Denton, E., Hanna, A., Amironesei, R., Smart, A., & Nicole, H. 2021. On the genealogy of machine learning datasets: A critical history of ImageNet. Big Data & Society 8(2). https://doi.org/10.1177/20539517211035955 Gibbs, M., Meese, J., Arnold, M., Nansen, B., & Carter, M. (2015). #Funeral and Instagram: Death, social media, and platform vernacular. Information, Communication & Society 18(3): 255-268. Papacharissi, Z. (2015). Affective Publics: Sentiment, Technology, and Politics. New York: Oxford University Press. Therieau, M. Vibe, mood, energy. The Drift, January 19. https://www.thedriftmag.com/vibe-mood-energy/