The artificial intelligence of a machine: Moving images in the age of algorithms

by Ruggero Eugeni and Patricia Pisters

In the Japanese anime series Psycho-Pass (2012-2013) citizens are all connected to a bio-mechanised collective mind that endlessly measures and scans their brains based on a numeric crime coefficient index. The Ministry of Welfare runs the Artificial Intelligence (AI) system. A colour-coded hue alerts law enforcers when criminal potentiality is deemed too high by the system. As we are writing this introduction in June 2020, in the midst of the pandemic in Europe, panoptic AI in the service of public health is no longer a speculative fiction due to measures to control COVID-19. In different countries around the world apps, coloured access codes, dashboards, and other digital tools have been introduced in various degrees of enforcement, providing Ministries of Welfare, other authorities, and big tech companies with an increasing amount of ‘intelligence’ about our physical health and whereabouts. Of course, we all know that AI has entered practically all domains of our lives and will continue to do so; the current crisis has only made the move to AI speedier. Universities all over the world collaborate with governments and corporations to build stronger AI units to analyse big data, to operate machines, to augment reality and create 3D learning environments. Many of these new AI applications will help in providing solutions for the challenges of the (near) future, such as controlling disease and the consequences of climate change. But all of this is not self-evident. In these new AI centres, there is sometimes an ethical committee that advises on the possible risks and benefits of specific applications.

With this special section of NECSUS, we want to present a selection of articles which address AI from a cultural perspective. Contemporary media culture is both engrained with speculative fictions about AI, but at the same time it is also increasingly engrained with and determined by artificial intelligence. And on all levels, a critical perspective is needed. The articles that we have selected operate on three levels in the AI discourse. First, we can see that film, television series, and video games confront us with several representations of both old and new ideas about non-human intelligence versus human intelligence. From this perspective we can observe both a certain continuity and some breaks. Indeed, on the one hand, the theme of the automaton and the artificial organism has been very present since the origins of cinema (think only of the cases of Metropolis, The Golem, Frankenstein, some of which are taken up in the essays in this section); such a topic is clearly rooted in the sensation of a decisive artificialisation of the subject’s life, typical of modernity and perfectly expressed by the experience of the cinema spectator. The staging of the relationship with the other as a living and thinking synthetic being runs throughout the history of cinema and the media, and is intimately linked to an uncanny and uneasy feeling (for example the Hal 9000 ‘character’ in Stanley Kubrick’s 2001 A Space Odyssey [1968]). More recently, however, this tradition has undergone a turning point: films such as Steven Spielberg’s A.I. (2001), Spike Jonze’s Her (2013), Alex Garland’s Ex Machina (2015), Denis Villeneuve’s Blade Runner 2049 (2017); television series such as Westworld (Jonathan Nolan and Lisa Joy, 2016-present) and (in various episodes) Black Mirror (Charlie Brooker, 2011-present), as well as some video games seem to deepen new emotional registers of the relationship with artificial subjects.[1] In particular, we highlight two of them: the subjective melancholic and saturnine feeling linked to the synthetic intelligence living experience; and the collective anger against the unequal rights and the racial bias penalising humanoids. We argue that these representations reflect a new cultural sensibility to the different types of AI: the fusion of natural and artificial components seems now taken for granted, and this overcoming opens up new kinds of personal and socio-political feeling concerning the ordinary coexistence with hybrid biological and technological forms. 

Many of the essays we have chosen address these issues. In his article on the myth of the Golem, Amir Vudka demonstrates how this Jewish myth can be seen as one of the earliest AI prototypes that translates both the anxiety and fascination for a human-made machine – a robot becomes independent of its master. Referring to the warning of AI experts in 2015 that we need to stay in control of our AI (something that is less than self-evident), the ancient myth of the Golem is more relevant than ever. Vudka’s contribution returns to the ancient Jewish inspirations of the myth and brings them back to classics of cinema, starting with Wegener’s Der Golem (1915), demonstrating that scientific inventions of machine thinking have always already impinged on a theological and spiritual dimension. Vudka further argues that ‘stories about AI combine to create a narrative ecosystem around AI that influences its research, reception, and regulation’. The contribution by Patricia Trapero does not look backwards to ancient myths of artificial intelligence but gives us a flash forward by looking at speculative fictions in three television series and a video game that all address possible worlds where AI has transformed into superpanopticons, but also have developed a certain degree of emotional intelligence. By examining both the modern-day Frankenstein-like creators of AI as well as including the point of view of androids, she questions what these narratives have to tell us about the construction of subjects and power relations that are worth taking into consideration. In his contribution, Martin Hennig takes a double perspective. He investigates how video games take up and extend much of the traditional discourse on technology and artificial intelligence as a conflict between human and machine intelligence, analysing how AIs are represented as either opponents, intrigants, or avatars; he also demonstrates how these discourses on AI in videogames are also embedded in the development of real AI in that they are implemented and experimented with in the game software itself.

From the level of representation, myth-making, and speculative fictions, we have then already moved to the machine vision and algorithmic interventions that address another level of problems and questions related to AI. This second perspective, referring to the mutual functioning dynamics between AI and the media, immediately leads us to highlight a very close reciprocal link between the two fronts. First, on the AI ​​front, we can notice that the new wave of interests and applications is strictly linked to the transition from rigid, rule-based algorithms of a logical-symbolic type (typical of the so-called GOFAI: Good Old-Fashioned AI) to flexible machine learning algorithms using randomness, probability, and approximation.[2] In turn, the success of this new generation of algorithms is closely linked to the availability of vast quantities of data from different sources, which are transformed in real-time and are capable of producing value (the four Vs of big data: volume, variety, velocity, and value).[3] Yet, this massive flow of data has become available only since the advent of social media, the traceability of user behaviour online, and the birth of the so-called digital, computational, networked or platform capitalism.[4] In other terms, the new generation of AIs has been able to take off only thanks to digital media, and in particular to the practices carried out online. Furthermore, the use of still and moving images captured by sensors and subjected to ‘datification’ has proved particularly useful in many areas of AI application, from surveillance (think only of face recognition) to defence (satellite images) and medicine (diagnostic images). Finally, the visualisation of the data processed by AI also benefits from know-how deriving from visual and audiovisual media. On the media front, the use of visual and audiovisual data processing solutions offered by AI has become extensive: machine vision intervenes in the processing of audio and visual data, from capture to editing; digital assistants and home automation interfaces use speech and visual recognition algorithms; AI automates the distribution of audiovisual products on SVOD and TVOD platforms such as Netflix or Amazon Prime, and modify the criteria for visibility and recognition of audiovisual products. If film scripts entirely written by AI is for the moment mainly a forecast, trailers produced by specialised software and editing programs assisted by AI such as Magisto are already a reality. 

Overall, we can affirm that the new generation of AI algorithms is eroding the boundaries between media and extra-media uses of moving images and sounds; consequently, we should perhaps rethink contemporary media from the point of view of a ‘political economy of light’, closely intertwined with a ‘political economy of digital data’. In other words, the blending between AI and media leads our reflections in two complementary directions: it pushes us to analyse the logics of production, circulation, and transformation of light (both from an optical-geometric and a physiological perspective[5]); also, it drives us to evaluate the intertwining of these logics with that of the production, circulation, and algorithmic transformation of digital data – without forgetting the principles of sovereignty that govern these dynamics and determine their trends.

Many papers we have selected directly address these issues. Federico Biggio first takes us back to the 1970s, to Gene Youngblood’s classic work Expanded Cinema, where he argued that a new type of cinema, expanded as media art, cybernetic movies, television-on-demand, and videospheres, is needed for a new kind of augmented (and even cosmic) consciousness. Biggio investigates three types of AI forms: augmented reality, machine learning, and search engine algorithms, in relation to three types of expanded gazes (increased, distant, and anaesthetic), to argue that the lesson to take from Youngblood is that the expansion of vision and the augmentation of consciousness go hand in hand, and that we need to read the developments of the increase of human intelligence with non-human tools always as a double process in culture as well as in science. Pei Sze Chow then takes us to the field of cultural entertainment by addressing AI applications in the (Hollywood) film industry. Via qualitative company case studies, she investigates how studios implement AI programs such as Scriptbook or Cinelytic at the greenlight stage of production. Her investigation addresses the issues of creativity and labour in the age of machine learning and questions the ethical implication of bias in respect to race, gender, and ideology when predictive analytic tools are replacing what has long been deemed a purely human intelligent line of work. These questions will grow in importance when AI programs such as Deep Story will enter the writer’s room and other departments of the audiovisual industry.

Questions of how AI intervenes in all kind of human activities, including the production of films, television series, and games have thus already led us to philosophical questions about the nature of intelligence in AI and the connected issues about creative labour and ethical concerns about the reproduction of bias and prejudice. At this third level, we must first consider that since its origins in the 1950s, works on AI have established a dialectical tension between two research trends. First, AI researchers tackled particular problems by identifying specific solutions which in turn expressed individual dimensions of mental processes: it is the so-called ‘weak AI’, which concentrates from time to time on perception, association, reasoning and decision-making, prediction, natural language processing, and so on. Furthermore, the work on AI expressed the theoretical need to reunify the different tools in an overall model that could account for the overall functioning of the human mind through its artificial reconstruction and simulation. This ‘strong AI’ can be defined as ‘a cross-disciplinary approach to understanding, modelling, and replicating intelligence and cognitive processes by invoking various computational, mathematical, logical, mechanical, and even biological principles and devices’.[6] Even today, we can find a tension between weak and strong AI, for instance in the quest for a ‘master algorithm’,[7] or the elaboration of models of Artificial General Intelligence.[8] However, compared to the efforts of thirty or forty years ago, at least two orders of novelty have taken over. First, it has become impossible to think of AI as purely logical and disembodied minds: logical processes have been increasingly integrated with probabilistic ones, while attention to emotional aspects of AI is sharply growing – for example through the recognition of emotions of human subjects based on the scanning of their faces, or through the modulations of the artificial voices in the Vocal User Interfaces (VUI) such as Apple’s Siri or Amazon’s Alexa. Furthermore, the active processes enacted by AI are becoming increasingly important: AI ​​is connected to sensors and effectors, so that they can act in and interact with the world around them: ‘More and more investigators are accepting the importance of situatedness and embodiment and exploring the degree to which a system’s interactions with a real-world environment are crucial.’[9] Not by chance, the most accredited AI university manual defines its object ‘as the study of agents that receive percepts from the environment and perform actions’.[10] Hence, the close link between AI and robotics, as well as the success of machine learning algorithms (typically reinforcement learning) which implement effective criteria of action and interaction of machines developing over time in complex and unpredictable environments. To sum up, cold and disembodied AI is taking over from warm and embodied AI, actively embedded in living environments.

Second, it has become impossible to think of AI as purely mental objects and processes, entirely split by the simulation of neural mechanisms. Many AI algorithms currently use connectionist or parallel distributed processing (PDP) models, composed by artificial neurons and synaptic connections inspired by the functioning of the human brain. It is interesting to consider that from a historical point of view these algorithms derive from research on the neural processes of visual perception: Mark Rosenblatt’s perceptron, which was created in the 1960s and is the ‘father’ of today’s artificial neural networks, inspired David Marr’s research on visual perception.[11] Still today, the machine vision and visual recognition processes constitute not only a substantial part of AI research, but a more general model for operating with artificial neural networks and their learning processes – for example with the multi-layer perceptron, the deep neural networks, and the associated learning processes.[12] Furthermore, these same algorithms are increasingly capable not only of analysing images and texts but also of creating new ones: thus, they manifest an AI ‘creativity’. This skill occurs both in ‘supervised’ forms, for instance in the work of many artists such as Hito Steyerl, Trevor Paglen, or Ian Cheng,[13] and in ‘unsupervised’ (or partially supervised) forms, for example the AIs that assist human subjects in creating images and videos in everyday aesthetic practices;[14] or in Generative Adversarial Network algorithms, that started creating artistic images to be used in art installations and successfully inserted in the art market.[15] Overall, this currently very active field of research calls for a series of reflections that are not only a prerogative of the philosophy of the mind[16] and neuroscience,[17] but which deeply involve film and media theory: what does it mean to ‘create’ a text, and what relationship exists between the creative processes and the ‘automatisms’ of a medium? What does ‘understanding’ mean, and in particular what relationship must be considered between ‘vision’, ‘understanding’, and ‘emotional intelligence’? Should we consider the possibility of ‘non-biological’ or ‘non-human’ forms of visual or aural understanding and feeling? Also, what relationship exists between human intelligence and ‘the intelligence of a machine’, quoting Jean Epstein’s classic take on the cinematograph?[18] How are AI models connected to those of cognitive neuroscience in accounting for the experience of the spectator? Particularly, how to reconcile an embodied conception of the moving image viewer with the process and ideas of ‘embodiment’ proper to recent AI research? And ultimately, what does AI (and mediated forms of AI) tell us about being human (viewers) in a post-human society?[19]

In particular, we have selected two papers that deal with this area of problems. Oshri Bar-Gil presents his findings on the case of the Google Clip camera, which promised to give its users an autonomous machine learning tool that would create a digital doppelganger, automatically filming and editing our lives. While the experiment was short-lived, Bar-Gil performed research within review blogs of the tools (as ‘netography’), analysing why this type of AI did not work. Furthermore, he addresses the more fundamental questions, whether it is possible and desirable to delegate our intentions to a digital self that, based on a predictive profile, can act on behalf of the user. Shintaro Miyazaki concludes our special section with a more activist approach in discussing how data-activist artistic collectives perform critical interventions in AI technologies themselves to expose the hidden dimensions of contemporary artificial intelligence. Anti-datamining, counter-algorithmicity, and algorithmic trading freak shows are some of the artistic interventions with pedagogical impact. We realise that with these seven contributions from the media theoretical domain we only address the tip of the iceberg of the challenges that the entanglements of human and non-human intelligence will bring us in the very near future where myths, speculations, and real inventions will translate into evolving questions about the expanded and transformed nature of intelligence. 

Authors

Ruggero Eugeni is Professor of Media Semiotics at the Università Cattolica del Sacro Cuore (Milano). He has been a visiting professor at Paris 3 and Paris 4 universities. He currently works on the relationships between media cognitive semiotics and neurosciences (and more particularly, from this perspective, on the subjective perception of time in the vision of moving images), and on the cinematic representations of hypnotism. For more information visit Media|Experience|Semiotics https://ruggeroeugeni.com/.

Patricia Pisters is Professor of Film, Media and Culture at the Department of Media Studies of the University of Amsterdam. She is one of the founding editors of NECSUS. In 2019 she was scholar in residence at EYE Filmmuseum Netherlands and research fellow at Cinepoetic at Freie Universitat Berlin. Her latest book New Blood in Contemporary Cinema: Women Directors and the Poetics of Horror is forthcoming in October 2020. For more information visit patriciapisters.com.

References

Alpaydin, E. Machine learning: The new A.I. Cambridge-London: The MIT Press, 2016.

Boden, M. Artificial intelligence: A very short introduction. Oxford-New York: Oxford University Press, 2018.

Bogost, I. ‘The AI-Art Gold Rush Is Here’, The Atlantic, 6 March 2019: https://www.theatlantic.com/technology/archive/2019/03/ai-created-art-invades-chelsea-gallery-scene/584134/ (accessed on 28 July 2020).

Domingos, P. The master algorithm: How the quest for the ultimate learning machine will remake our world. New York: Basic Books, 2015.

Dyer-Witheford, N., Kjøsen, A., and Steinhoff. J. Inhuman power: Artificial intelligence and the future of capitalism. London: Pluto Press, 2019

Epstein, J. The intelligence of a machine. Minneapolis: Univocal, 2014 (orig. in 1946).

Frankish, K. and Ramsey, W. ‘Introduction’ in The Cambridge handbook of artificial intelligence, edited by K. Frankish and W. Ramsey. Cambridge: University Press, 2014: 1-11.

Fry, H. Hello world: How to be human in the age of the machine. London: Transworld, 2018.

Goodfellow, I., Bengio, Y., and Courville, A. Deep learning. Cambridge-London: The MIT Press, 2016.

Kelleher, J. and Tierney, B. Data science. Cambridge-London: The MIT Press, 2018.

Kitchin, R. The data revolution: Big data, open data, data infrastructures, and their consequences. Los Angeles: Sage, 2014.

Manovich, L. A.I. aesthetics. Moscow: Strelka Press, 2018.

Marcus, G. and Davis, E. Rebooting A.I.: Building artificial intelligence we can trust. New York: Pantheon Books, 2019.

Mayer-Schönberger, V. and Cukier, K. Big data: A revolution that will transform how we live, work and think. London: John Murray, 2013.

Obrist, H. ‘Making the invisible visible: art meets A.I.’ in Possible minds: Twenty-five ways of looking at A.I., edited by J. Brockman. New York: Penguin Press, 2019: 206-218.

Russell, S. and Norvig, P. Artificial intelligence: A modern approach, third edition. London: Pearson, 2016. 

Sejnowski, T. The deep learning revolution. Cambridge-London: The MIT Press, 2018.

Tegmark, M. Life 3.0.: Being human in the age of artificial intelligence. New York: Alfred Knopf, 2017.

Togelius, J. Playing smart: On games, intelligence, and artificial intelligence. Cambridge-London: The MIT Press, 2018.

[1] Togelius 2018.

[2] Alpaydin 2016.

[3] Mayer-Schönberger & Cukier 2013; Kitchin 2014; Kelleher & Tierney 2018.

[4] Dyer-Witheford & Kjøsen &Steinhoff 2019.

[5] Elsaesser 2016, pp. 376-383.

[6] Frankish & Ramsey 2014, p. 1.

[7] Domingos 2018.

[8] Boden 2018, pp. 18-49.

[9] Frankish & Ramsey 2014, p. 9.

[10] Russel & Norvig 2016, p. viii.

[11] Sejnowski 2018, pp. 37-78.

[12] Goodfellow & Bengio & Courville 2016

[13] Obrist 2019.

[14] Manovich 2018.

[15] Bogost 2019.

[16] Boden 2018.

[17] Markus & Davis 2019.

[18] Epstein 1946.

[19] Tegmark 2017; Fry 2018.