INT J COMPUT COMMUN, ISSN 1841-9836 Vol.7 (2012), No. 4 (November), pp. 606-616 The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis A.M.P. Brasoveanu, I. Dzitac Adrian M.P. Brasoveanu MODUL University Vienna, Department of New Media Technology, Austria, 1 Am Kahlenberg, 1190 Vienna, E-mail: adrian.brasoveanu@modul.ac.at Ioan Dzitac 1. Aurel Vlaicu University of Arad, Romania, 310330 Arad, 2 Elena Dragoi, and 2. Agora University Romania, 410526 Oradea, 8 Piata Tineretului E-mail: rector@univagora.ro Abstract: As semantic multimedia is approaching mainstream, even the great im- provements that can be seen in its classic schools, like the data mining inspired In- formation Retrieval based on metadata analysis, or Computer Vision, might not be enough. We identify a new group that gains traction in the semantic multimedia community and which uses as starting point developments from psychology and vi- sual communication. For the purposes of this article we restrict our domain to visual rhetoric as we consider it to yield the biggest potential for future developments. Living in times when the periods between crises seem to be shorter and shorter, we look at how developments in semantic multimedia can be used for predicting and overcoming crises. We analyze at least 2 aspects related to this: using information visualization to understand the evolution of crises and creating multi-layered semantic multimedia technologies that can easily be adapted to use a variety of sources and solve problems from different domains. In both cases we show how techniques inspired by visual rhetoric (information linking, framing, composition) in conjunction with named entity recognition offer a lot of benefits. The section related to multi-layered semantic multimedia technologies also draws on the lessons learned while designing a prototype application aimed at improving tourism decision making process. The article ends with a discussion on evaluation methods for multi-layered semantic technologies applications. We look at how to evaluate them on both levels: mech- anisms (information linking versus raw named entity recognition when generating visuals, for example), and decision making strategies (Do such systems actually solve real problems related to crises, create jobs or at least can they be repurposed to solve other problems than the one with which we have started?). Keywords: semantic multimedia, visual rhetoric, text to visual matching, interactive documentary, crisis strategies, multimedia storytelling "The progress of civilization can be read in the invention of visual artifacts, from writing to mathematics, to maps, to printing, to diagrams, to visual computing." Stuart Card et. al. - Readings in Information Visualization 1 Introduction In today’s fast Web there is a need for elegant mechanisms that can help us understand how to process, store, retrieve and present the huge amount of information contained in multimedia files. Semantic multimedia [23] [24], the branch of Semantic Web focused on the analysis of mul- timedia documents, traditionally employed methods like metadata analysis, feature extraction Copyright c⃝ 2006-2012 by CCC Publications The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis 607 or multimodal analysis, and as a result, the core researchers were split in two large groups: a group that had its roots in text processing and data mining, and a group with roots in computer vision [21] [23] [24]. Of course at times one will need to apply both methods to get meaningful information from a multimedia system. Today, new groups are slowly emerging, driven by the advances from fields like visual communication, psychology or biology. The group at which we adhere is still a small one and tries to tie the ideas from the modern visual rhetoric to informa- tion visualization, multimedia processing and generation, interactive documentaries and other semantic multimedia areas. Visual rhetoric [11] is one of the new disciplines taken into account by the semantic mul- timedia researchers. We consider this cross-pollination natural since both fields deal with the interpretation of visual media (paintings, photographs, movies, games, etc.). The question we would like to address in this paper is how can we design semantic technologies that take into account the findings from visual rhetoric? The rest of the paper is organized in 4 sections. Section 2 starts with a discussion about the various schools and definitions of visual rhetoric; and identifies several ideas that have a potential for growth in the field of semantic multimedia. It also contains a review of the related work. Section 3 continues with a short analysis of the role that visual methods have in decision making with a special focus on crisis economics. Section 4 presents some ideas about prevention of crises and a case study built around a prototype application. We conclude our paper with a discussion on the various evaluation methods that can be used in order to assess the success of our enterprise, both on the level of the mechanisms described (comparisons between different methods for generating multimedia content) and on the level of decision making strategies. 2 The Role of Visual Rhetoric in Semantic Multimedia and Re- lated Work Images, video files or graphics of any kind (paintings, infographics, interactive visualizations) always tell more than we would like to admit. Since the biggest processing engine we have is our brain we should pay more attention to how we process and present any information using multimedia channels. Visual rhetoric is one of the modern disciplines that can help us do precisely this, if we take the time to study it and apply it to our representation and interactivity problems [11]. While visual rhetoric is not new, its theoretical treatment and multimedia applications are. The seeds of this discipline can be found in the articles about art theory, film art and iconology, published in the German Space since the third decade of the 20th century. Probably the most famous exponents of that period were Rudolf Arnheim [2] and Ernst Gombrich [9]. The term visual rhetoric was rarely used at the time, but most of the elements discussed in their essays (from equilibrium to lighting or color, but also space or dynamics) are included in modern treatments of visual rhetorics. The influence of this movement goes well beyond visual communication, and their ideas can be found almost everywhere from architecture to game design. This wave was mostly focused on issues of representation and composition in art. During the ’60s and ’70s there was a French wave of visual rhetoric, which was inspired from the film criticism of Cahiers du Cinema, literary criticism and philosophy, its most famous exponents being Roland Barthes [3] and Jacques Bertin [4]. Cinema, photography and charts were the focus of the essays published during this wave. Today, the dominant current in visual rhetoric theory is Anglo-Saxon, Gunther Kress [17] or Charles Hill [11] being some of its most respected proponents. Eric Kandel [16], belongs probably to both the German and Anglo-Saxon wave (he left Vienna when he was 9 years old in 1938, but he always tried to keep contact with fellow Austrians like Ernst Kris or Ernst Gombrich who were 608 A.M.P. Brasoveanu, I. Dzitac influential in establishing the grounds on which a scientific theory of visual rhetoric will someday be formed). The current wave is one of solid grounding, multimodality, mathematics, computer science and semiotics being used to connect the dots between the various long running threads. Complex questions are asked (questions like: What is the role of music in a certain scene from a movie? How can certain elements be used in the same scene to enhance its meaning?). By doing this, the current wave starts to deconstruct the authorial intentions in a scientific manner. While there is no single all-encompassing definition of visual rhetoric to this day, there is a consensus regarding the fact that you can create a visual rhetoric space for any discipline. Some of the fields where visual rhetoric can and should be used are investigated in [11] together with the possible definitions of visual rhetoric as seen from those fields of study. Basically all definitions agree on one aspect: visual rhetoric is a form of communication that uses images to construct meaning or arguments. By extension, visual literacy defines the way we respond to images and it implies that we are already well trained in how to read images. The beginnings of visual rhetoric were controversial (as proved by the Ernst Haeckel biological images forgery case [8]), but its continuous improvement, especially during the last two decades, has led to its acceptance as one of the leading areas of research in visual communication. It is enough to look up the list of publications from the premier venues for semantic web, multimedia or information visualization publications (Journal of Web Semantics, ACM Multimedia, IEEE Multimedia, IEEE TVVG, ACM TOMCCAP) during the last years (2008 - 2012) to discover that some of the articles that received a lot of attention (Best Paper Awards, quotations, discussions in other papers, even sequels) apply ideas inspired by visual rhetoric like: visual query suggestion [30], narrative visualization [20], affective image classification [18], framing effects [13] and color naming models and their applications [10]. Hullman and Diakopoulos [13] apply their ideas on visualization rhetoric to a class of visualizations identified by Segel and Heer [20] as narrative visualizations. Narrative visualizations do not just visually present some numbers, but also draw attention to the story behind those numbers, and in doing so they need to deploy an entire arsenal of techniques like provenance rhetoric, mapping rhetoric (visual metaphors, contrast, etc), linguistic or procedural rhetorics [13]. The paper about color naming models [10] is important mainly for library builders (especially JavaScript libraries), while [18] uses features generally used in psychology and biology to create an affective image classification. Some of the metrics used in [18] are color (name, contrast, features), texture (wavelet, Tamura, etc.), composition (depth of field, rule of thirds) or content (human faces, skin). An interesting problem in our view is that of using images to illustrate arguments. An existing approach towards this problem involves finding the entities from texts (or even a search box) and associating them with images from medical literature (using caption processing, image processing, and topic discovery), and can be found in the Structured Literature Image Finder [1]. The training data used manually annotated images from different subcellular locations. SLIF, developed at CMU, is basically a restricted search engine, one that has as domain medical literature. For medical texts software like this is extremely valuable, as it is often the case that an image presents an entire story. Identifying images that best represent entities is a challenging task by itself, but identifying images that would best illustrate a point of view is an even more daunting task. It is however the task that anyone involved in visual rhetoric would like to solve, especially people involved in advertising. Bocconi, Hardman and Nack [5] were able to generate matter of opinion docu- mentaries by "framing" into a larger conversation clips from interviews that captured people’s reactions to the September 11 terrorist act. After proving how to build the "framing" mecha- nisms using semantic graphs, they conclude that the role of visuals in providing support for the subject matter in documentaries needs to be further developed. We used this assumption as a The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis 609 starting point for our investigations and discovered that there are several theories in social semi- otics [11] [17] [27] dedicated to supporting verbal meaning with visual artifacts, and we started to build a framework around these theories. What we noticed by reviewing the literature is that while these articles are hardly related when it comes to subject (apparently there aren’t many connections between visual query sugges- tion, information visualization or color naming models), and some of them do not even mention the term visual rhetoric (except for [13] which refers to "visualization rhetoric"), almost all of them quote some of the pioneers of the field (Rudolf Arnheim [2], Johanes Itten [14], Roland Barthes [3], or Jacques Bertin [4], for example), and some were influenced by the research group led by Alberto del Bimbo [7]. This suggests that a third group is gaining lots of traction in the area of semantic multimedia, and that this group embraces theories from art, psychology or biology. The surveyed papers are built upon the philosophy of the early pioneers of the field, while we base our work on theories developed in the last 15 years. 3 Visual Rhetoric for Decision Making in Times of Crisis Mainstream economists almost always fail to predict crises, and the example of the current crisis (started in 2008 with the collapse of the American housing market, and at the time of writing - 2012 - still unfinished) is one of the best. There a few economists who are said to have predicted this crisis (Nouriel Roubini, Peter Schiller, Nassim Taleb and others) [34], but there are hardly any graphics that prove their theories. The most interesting theories present dragon-kings (significant or meaningful outliers), black swans (in essence events that are almost impossible to predict) and other models that would lead to accurate predictions. Dragon-kings models proposed by ETH’s Didier Sornette [15] [22] [31] do not involve events that can’t be predicted, as black swan events suppose. Sornette’s group anticipated some short bubbles and published the results sometimes few days before the actual events occurred [31]. Other economists just improved SOM models and run them against datasets related to the current crisis [19]. While they did not predict a new crisis or the length of the current one with a great accuracy, these models are still useful and can help us understand what happens in today’s high frequency trading markets. We think that the worst part when it comes to crisis prediction or visualization is that even some of the best visualizations (take the visualizations from one of the top contests, for example [33]) do not present us with the visual cues (dragon-kings, black swans) that would make us easily understand what happened during the last years. Just plotting some data without highlighting the events that suggest bubbles or other crisis scenarios is not going to help us too much. We have to spend more time thinking about the results of the visualizations and how to present those results in such a way that they are easy to understand. This means more time spent tackling the problems related to manipulation and bias, since graphics can in the same time improve our understanding of current or past events, but they can also be used to manipulate the public opinion. In today’s connected world, where most of the governments put their data online (including financial data), we think that this dual nature of visualization rhetoric is something we must carefully address. [13] shows how to "frame" narrative visualizations to present different points of view related to the same event (poll predictions). Another method to insert more information into visualizations would be to combine various visual metaphors as demonstrated in [12]. We think there are multiple reasons why current visualizations do not take into account such phenomena. First, many visualizations are not done by interdisciplinary teams, so if the researchers have not heard about such events (dragon kings, black swans, etc) they will not be represented in the 610 A.M.P. Brasoveanu, I. Dzitac end product (this assumption does not apply to bigger outlets with traditions in visualization like The New York Times or Guardian). Second, most of the visualization rhetoric used today still comes from several sources (usually Jacques Bertin [4], William Cleveland [6], Edward Tufte [25] [26], Leland Willkinson [29], the last one being more recent than the rest) mostly focused on visual presentation of data, but not on framing and the consequences of framing and story selection. Third, there is no single, uniform, easy to understand and use visual rhetoric for any type of decision making. This means that all visualization designers need to master visual literacy. For today’s visualization designers there is an easy path towards mastery of both visual rhetoric and visual literacy: they can start learning online (outlets like: [36]- [41]) and then go on and read the masters (classics: [4], [6], [25], [26], [29], or modern: [42]- [44]). Just by choosing a story and applying some creative layering techniques over a plot we will not be able to predict and prevent a crisis. If we want to be able to do that, we will have to rely on other mechanisms like job creation, for example. Or we will have to create technologies that work on multiple levels and are easy to adapt for solving different problems. Prevention is not going to be of much help as long as the visualizations we use do not show us the dragon-kings or black swans. 4 Multi-layered Strategies for Overcoming Crisis - Semantic Tech- nologies to Increase Profits A good case study for our ideas would involve a situation in which we can produce multimedia content in order to leverage some of the advantages that interactive visual environments have over traditional media. Tourism industry, entertainment, or politics could be considered some of the premier venues where image is everything and such applications would help a lot. Take for example this scenario: you have several videos in which speakers tell stories about what they like in a city. If you want to create a short movie from this clips, you will likely want to convince those who watch it that the city being spoken about is really beautiful. This means you will have to replace in many places the images of the speakers with those of the objects or concepts being spoken about. Using visual rhetoric is just one of the best ways to automate this process. Our approach towards the use of visual rhetoric in semantic multimedia is based on some of the more recent findings, like [11] and [17]. The main goal of our project is to understand how to use visual rhetoric in multimedia environments. Some secondary goals for the project are to use textual to visual matching methods and visual content generation methods. Most of the work presented in Section 2 uses visual rhetoric only for interpretation/explanation purposes (this visualization presents the evolution of a player during the last season, for example), whereas our goals are more inclined towards the possible applications of visual rhetoric in the space of multimedia content generation (automated movie generations, automated footprint generation, automated summary generations). The case study involves a prototype web application called Interview eXplorer and used to generate visuals for a series of interviews with people (mostly students) regarding their lives in a foreign city (Vienna, the capital of Austria). We selected Vienna as the city that must be explored, not only because we spend a lot of time in the city, but also because: • It is the city where the whole movement with integration between science and art started around 1900, according to the Nobel Prize laureate Eric Kandel [16]; • It is a city with lots of historical sites which are well represented on any media property, from Twitter to Wikipedia (Belvedere or Stephansdom, for example); • It is a big city with enough green spaces (Donauinsel, Vienna Woods, etc.); The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis 611 Figure 1: The components of the prototype • The landscape changes quite fast because there are lots of new construction sites; • While it does not have so many iconic images like London, Paris or New York, it consistently ranks higher on Mercer’s Quality of life tops for few years now [36] (which can make for interesting life questions which should theoretically be harder to illustrate through images). The generated short documentaries (webisodes) can be used in effective marketing campaigns for attracting students or tourists, for example, but they can also be used as research tools in economy or social sciences in order to understand the needs and the habits of the population from a certain area. The interviews contain questions related to the parts of Vienna that attract the students (buildings, parks, Danube - easier to illustrate if we use an automated approach based on named entity recognition), but also questions about how well they integrated into the new environment (work, friends, social life - harder to illustrate even if you use a manual approach, and it gets even worse with automated approaches). The application can function in 2 modes: Search (where we can just see the proposed visuals and some widgets with additional information about the entities mentioned) and Explore (where we can see proposals on how to illustrate episodes with the related identified visuals). For additional information we integrated widgets that display data from Wikipedia or Twitter, or the maps from Google, for example. Integrating Facebook widgets was problematic, as some of the people interviewed felt that it would be a serious breach of their privacy. They were basically not aware that all you need to find a person on Facebook is a name. People that had Twitter accounts were more likely to agree to display their information on the Twitter widget, because they see this as free publicity. Currently, all the videos, images and interviews used in the prototype are from the personal collection of the author. A long term goal of the project would be to use any free images or videos available on the web that are related to the topic discussed. The front end of the prototype was programmed in JavaScript, while the back end uses several other programming languages. The communication between different components is done using the JSON format. Since this is a prototype, and not a commercial application, we are not able to provide metrics regarding its profitability. The expression from the title of this section should be interpreted as: since semantic technologies are now cheaper than they used to be, they can easily be used to 612 A.M.P. Brasoveanu, I. Dzitac Figure 2: Interview Explorer Prototype - First Iteration Figure 3: Visual suggestions tab from the eXplore mode. This simple visualization offers only several suggestions for replacing the frames of each video block. The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis 613 create applications that provide lots of features at an accessible price. Maybe the most interesting aspect related to the development of such applications is the fact that you can create a stack of technologies to solve a specific problem (replace the image of the person that is interviewed with images about the things he is talking about) and easily repurpose them to solve other problems (like use the generated documentaries for marketing purposes, or add an interactive visualization layer on top and provide data that can be useful for government to understand foreign citizens living in Austria, for example). Seen from this perspective, the fact that semantic web is now becoming mainstream should help a lot of companies create the tools that will help them easily navigate through the bad periods they are now confronting. 5 Discussion and Future Work By looking at the work presented in Section 2, it can be easily seen that there is a new current in semantic multimedia. This current might not yet be on par with Information Retrieval or Computer Vision, but it is certainly developing into something powerful and useful in the same time. It might not be a fully formed school like the ones we mentioned, but given the large number of papers in top venues we think that this school will be important in the next couple of years. Transforming visual rhetoric into science is not something that will happen overnight. It takes time to build the mathematical models and the simulations that are associated with scientific processes, and also needed in order to reproduce the results. As we have seen in Section 3, many crisis prediction models should also include a visual component (SVM visualization, dragon-king, black swan or something else). We think that visualizations should take this finding into account and such patterns should be discovered as soon as possible. It is hard for us to understand why big companies invest millions or billions of dollars or euros into visualization systems for real-time stock trading, but when it comes to presenting their findings to the public they almost never show the patterns that could lead to crises. Before fractal analysis we could argue that there were no good mathematical models to predict crises [15] [22] [31], but now since we have such models we should use them. The prototype from Section 4 is going through new iterations. Future work will involve trying to replace images of speakers with images of the concepts being spoken about. We will also add new layers to our architecture: a visualization layer, a sentiment analysis layer, and so on. We are currently undertaking an evaluation of the system on components level. It is currently the only method to evaluate it since our system uses images and videos that are not from standardized datasets like TRECVID [35]. We also do user evaluations against each component, because it is important to know what the users feel about the end result. First reactions of the interviewed persons were that the system looks good and it is useful. They would even see themselves using it, if it would be open to public, since they consider it has the potential to be great in the space of personalized entertainment. We will perform a more detailed survey to assess the strength and weaknesses of our approach in the eyes of the users. Since today cinema viewers are accustomed to seeing complex narratives, we know that their expectations are high. For example, the ending from one of last year’s most critically acclaimed movies: Tinker, Tailor, Soldier, Spy mixes scenes that happened during at least 4 periods of time (the distance between each being several years) in only 5 minutes (the focus is on the relationship between two characters, and the ascension of the third in light of the recent events), but still manages to keep us involved and provide a satisfactory and artsy conclusion in the same time. We are aware that reaching such an artistic mastery requires a long commitment from our side, but our first target is not Tinker, Tailor, Soldier, Spy, when it comes to generating an engaging narrative (as that movie is also a dramatic adaptation of a novel), but rather The Autobiography of Nicolae Ceausescu, a documentary which manages to 614 A.M.P. Brasoveanu, I. Dzitac tell the dictator’s story without any background narration. From our point of view reaching the artistic and information complexity level of this documentary (editing, scoring, narrative comprehension, etc.) will take several years, but the first steps toward this goal have already been made. Assessing the success of our prototype system on the level of decision making strategies represents a complex process which is also likely to take several years. It involves developing future versions and showing the end product to potential customers in order to draft a commercial version in one day. A commercial version might easily function on multiple levels as we suggested. For example, if we would implement it for a touristic city portal, the system could generate both the ads that could be served to visitors (ads for ski during winter, and for hiking during summers), but also the presentations for various locations from the city and webisodes that show different aspects of living in that community. Combining such systems with social media monitoring for example would allow us to extract more information from chained events like the Arab Spring. This will offer us an unprecedented level of access to information to real historical events, which is a thing all historians would like. Adding powerful prediction models to such a system would make it the ultimate crises prediction and intervention tool. We are only a few steps away from such systems as they are predicted in movies like Minority Report. Bibliography [1] A. Ahmed, L.P. Coelho, A. Arnold, J. Kangas, A.B. Sheikh, E. Xing, W. Cohen, R.F. Murphy, Structured literature image finder: Parsing text and figures in biomedical literature, Journal of Web Semantics: Science, Services and Agents on the World Wide Web, 8 (2-3), 151-154, 2010 [2] R. Arnheim, Art and Visual Perception: A Psychology of The Creative Eye, 2nd Edition, Cambridge, 2004. [3] R. Barthes, Image, Music, Text, Hill and Wang, 1977. [4] J. Bertin, Semiology of Graphics: Diagrams, Networks, Graphs, 2nd Edition, ESRI Press, 2010. [5] S. Bocconi, F. Nack, L. Hardman, Automatic generation of matter-of-opinion video documen- taries, Journal of Web Semantics, 6 (2), 139-150, 2008. [6] W.S. Cleveland, The Elements of Graphing Data, 2nd Edition, Hobart Press, 1994 [7] C. Colombo, A. Del Bimbo, P. Pala, Semantics in Visual Information Retrieval, IEEE Mul- timedia, 6(3): 38-53, 1999. [8] P. Dombrowski, Ernst Haeckel’s controversial visual rhetoric, Technical Communication Quarterly 12: 303-319, 2003. [9] E.H.Gombric, Art and Illusion. A Study in the Psychology of Pictorial Representation, Mil- lennium Edition, Princeton University Press, 2000. [10] J. Heer, M. Stone, Color Naming Models for Color Selection, Image Editing and Pallette Design, ACM CHI, May 5-10, 2012, Austin, Texas, USA. [11] C.A. Hill, M. Helmers, Defining Visual Rhetorics, Lawrence Erlbaum Associates, London, 2004. The Role of Visual Rhetoric in Semantic Multimedia: Strategies for Decision Making in Times of Crisis 615 [12] A.Hubmann-Haidvogel, A.M.P. Brasoveanu, A. Scharl, M. Sabou, S. Gindl. Visualizing Contextual and Dynamic Features of Micropost Streams, In Proceedings of the WWW’12 Workshopon on ’Making Sense of Microposts’. Lyon, France, April 16, 2012, CEUR Workshop Proceedings Vol-838, 34 - 40, 2012. [13] J. Hullman, N. Diakopoulos, Vizualization Rhetoric: Framing Effects in Narrative Visual- ization, IEEE TVCG, Vol. 17, No. 12, 2011, 2231-2240. [14] J. Itten, The Art of Color: The Subjective Experience and Objective Rationale of Color, John Wiley, New York, 1973. [15] Z.-Q. Jiang, W.-X. Zhou, D. Sornette, R. Woodard, K. Bastiaensen, P. Cauwels, Bubble di- agnosis and prediction of the 2005-2007 and 2008-2009 chinese stock market bubbles, Journal of Economic Behavior & Organization 74 (3), 149-162, 2010. [16] E. R. Kandel, The Age of Insight: The Quest to Understand the Unconscious in Art, Mind, and Brain, from Vienna 1900 to the Present, Random House, 2012 [17] G. Kress, T. van Leeuwen, Reading Images, Routledge, London, 2006. [18] J. Machajdik, A. Hanburry, Affective Image Classification using Features Inspired by Psy- chology and Art Theory, Proceedings of the ACM Multimedia, October 25-29, 2010, Firenze, Italy, 83-92. [19] P. Sarlin, D. Marghescu, Visual Predictions of Current Crises: A Comparison of Self- Organizing Maps with Probit Models, TUCS Techical Report, No 978, June 2010. [20] E. Segel, J. Heer, Narrative Visualization: Telling Stories with Data, IEEE TVCG, Vol. 16, 2010, 2231-2240. [21] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, R. Jain, Content Based Image Re- trieval at the End of the Early Years, IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 22 (12), page 1349-1380, 2000. [22] D. Sornette, Dragon-kings, black swans and the prediction of crises, International Journal of Terraspace Science and Engineering, 2 (1), pp. 1-18, 2009. [23] S. Staab, A. Scherp, R. Arndt, R. Troncy, M. Gregorzek, C. Saathoff, S. Schenk, L.Hardman., Semantic multimedia, Reasoning Web, 4th International Summer School, Venice, Italy. Volume 5224 of LNCS, Springer, 125-170, 2008. [24] R. Troncy, B. Huet, S. Schenk, Multimedia Semantics. Metadata, Analysis and Interaction, Wiley 2011. [25] E. R. Tufte, The Visual Display of Quantitative Information, 2nd Edition, Graphics Press, 2001. [26] E. R. Tufte, Beautiful Evidence, Graphics Press, 2006. [27] T. van Leeuwen, Introducing Social Semiotics, Routledge, 2005. [28] W.-N. Wang, Y.-L. Yu, Image Emotional Semantic Query Based On Color Semantic De- scription, Proceedings of the Fourth International Conference on Machine Learning and Cy- bernetics, Guangzhou, 18-21 August 2005, p.4571-4576. 616 A.M.P. Brasoveanu, I. Dzitac [29] L. Wilkinson, The Grammar of Graphics, Second Edition, Springer, 2005. [30] Z.J. Zha, L. Yang, T. Mei, M. Wang, Z. Wang, T.S. Chua, X.S. Hua, Visual Query Sugges- tion: Towards Capturing User Intent in Internet Image Search, ACM TOMCCAP 6, Issue 3, 1-19, 2010. [31] W. Yan, R. Rebib, R. Woodard, D. Sornette, Detection of Crashes and Rebounds in Major Equity Markets, http://adsabs.harvard.edu/abs/2011arXiv1108.0077Y, last accessed: 8.04.2012. [32] http://mazamascience.com/WorkingWithData/?p=870, last accessed: 08.04.2012. [33] http://www.informationisbeautifulawards.com/, last accessed: 08.04.2012. [34] http://www.economicpredictions.org/who-predicted-the-financial-crisis.htm, last accessed: 08.04.2012. [35] http://trecvid.nist.gov/, last accessed: 08.04.2012. [36] http://www.mercer.com/articles/quality-of-living-survey-report-2011, last accessed: 12.07.2012. [37] http://www.visualisingdata.com/, last accessed: 12.07.2012. [38] http://www.informationisbeautiful.net/, last accessed: 12.07.2012. [39] http://flowingdata.com/, last accessed: 12.07.2012. [40] http://www.visualcomplexity.com/vc/, last accessed: 12.07.2012. [41] http://visual.ly/, last accessed: 12.07.2012. [42] http://eagereyes.org/, last accessed: 12.07.2012. [43] http://blog.typekit.com/2012/06/12/designing-data/, last accessed: 12.07.2012. [44] http://twitter.com/nytgraphics/, last accessed: 12.07.2012.