Screening the financial crisis: A case study for ontology-based film analytical video annotations

by Jan-Hendrik Bakels, Matthias Grotkopp, Thomas J.J. Scherer, and Jasper Stratil

AdA Filmontology annotation dataset with DOI 10.5281/zenodo.8328663 on Zenodo.

Project description and background

The film-analytical semantic ontology and research data introduced in this paper have been developed and gathered over the course of four-and-a-half years (2016-2021) in the context of the project Audio-Visual Rhetorics of Affect (AdA^[1]).The project’s aims were two-fold: a) developing a systematic, computational approach to analysing audio-visual aesthetics and thereby b) adding another empirical level to a previously developed qualitative methodology (eMAEX) aimed at assigning affective qualities to spatio-temporal patterns of audio-visual composition.^[2] By publishing the generated data, we hope to contribute to a culture of discussion and cooperation on analytical research in the field of digital film and media studies.

The explicit interest in systematically annotating and qualifying spatio-temporal patterns derived from theoretical and methodological research within film studies that combines film-phenomenological as well as film-aesthetical perspectives.^[3] This research follows the basic assumption that the process of meaning making on the side of viewers draws on the embodied, sensuous experience of audio-visual movement in the broadest sense (e.g. cutting rhythms, camera pacing, choreographies, framing, music, et cetera) – and the assumed affective dynamics of particular expressive figurations of movement – as much as on dimensions more commonly associated with semiotics and understanding (e.g. dialogue, symbolism, represented actions, et cetera). Within the preceding eMAEX-framework these dynamics had been grasped by means of free text descriptions of audio-visual dynamics that were formalised on the basis of instructions for authors.^[4] The AdA project, in turn, aims at basing this process of description and qualification on visualisations of patterns developing across different levels of audio-visual composition (resembling the principle of a musical score^[5]) by means of video annotation software.

Against this backdrop, we tried to identify a field of audio-visual communication in which the use of affective reasoning is commonly accepted as evident. Audio-visual discourses on socio-political crises provide an interesting research domain because they usually combine different perspectives on causes, characteristics, and societal impacts with a strong rhetorical urge to promote solutions or ways of moving forward within highly polarised controversies (i.e. brought forward by different communities and actors). The project chose to focus on feature films, documentaries, and television news segments on the global financial crisis (2007-) aiming at qualifying, comparing, and relating different rhetorics of affect (i.e. the specific affective qualities grounded in particular patterns of audio-visual composition). The decision to conduct a larger corpus study across different types of audio-visual media was motivated by the intention to create a study set-up that would inherently ask for a cooperative and collaborative approach to film analysis.

An additional aim of the project was to offer film analytical tools and data that could also be used in other fields of research such as communication studies, political science, history, et cetera. Through the formalisation of analytical expertise from film studies we hope to facilitate the consideration of qualitative aesthetic dimensions in addition to the often prevalent use of quantitative approaches and content analysis. In order to achieve this, the project first had to enhance existing video annotation software in two ways: a) by developing an annotation template including systematic vocabulary for different aesthetic levels as well as possible attributions (in computational terms: values); b) by developing a software approach to visualising annotations across different levels following the idea of the audio-visual ‘musical’ score. In order to enhance the gathered data’s potential for collaborative and interdisciplinary research as well as enabling film analytical research from single case studies up to larger processes of machine learning, we chose to publish the project data as Linked Open Data using the semantic web standard RDF.^[6]

Video annotation in the digital humanities

With the vocabulary, the annotation data, and the semantic web technologies, we are openly appealing to a wide scope of various disciplines and communities with different knowledge bases as well as different corresponding usage scenarios. This is a reaction to the precarious position of audio-visual data within the heterogeneous field of digital humanities, which is largely dominated by approaches to language and written texts, digitised physical artefacts and collections, and methods like the reconstruction of actor networks.^[7] For film and media studies this results in an emphasis on film historiography that focuses on biographical data,^[8] cinema venues, exhibition and distribution,^[9] or rendering paratexts and sources accessible.^[10] In other words, digital film studies is mostly concerned with data about the production and circulation of films rather than analytical data gleaned from films.

The audio-visuality of film and video however has a difficult stance in this respect, with some notable exceptions and pioneers which this project builds upon (see endnotes 11-18). This has mainly two reasons: first, from a computational point of view, the size of digital video data is by many orders of magnitude larger in comparison to text and metadata; second, the qualitative, subjective dimension of an embodied, temporal, and multimodal viewing experience is extremely difficult to model as a data structure. One solution is to start from the other end, by looking only for the parameters that are blatantly open to quantification (like shot lengths and their statistical evaluation^[11]), using specific algorithms for specific tasks (like colorimetrics^[12]), doing artistic research dealing with visualisation and sonification,^[13] or even to combine different algorithms for proxy reasoning (such as face recognition relative to the image in order to express field sizes^[14]).

The aim to preserve the temporal and multimodal dimension of synchronous and diachronous relationships across different levels of visual, auditory, and narrative phenomena^[15] led to many parallel developments in digital film studies regarding annotation tools (ELAN,^[16] Lignes de temps^[17]). From the start of our project we have been involved in a dense community of exchange and advice within the field (especially with Barbara Flückiger’s team working on the VIAN-platform^[18] focusing on film colours). One central aspect that we identified was the need to find ways not just to export and import data between different software tools but to make these data meaningful in different environments. In addition, while exchanging discrete sets of annotation data between two project teams is one thing, to have a commonly accepted convention for a structured mark-up language for audio-visual data that goes beyond heterogeneous lists of labels is quite another.^[19] In other words, the latter would be the film studies equivalent to the work of the TEI or MEI consortium.^[20] Moving annotation data between applications in order to benefit from their interfaces for different algorithms for the automatic classification of visual and acoustic events, different forms of display, or for the statistical evaluation can only be reliably done on a larger scale with a machine-readable data framework capable of linking the different vocabularies.

AdA Filmontology & Advene – The framework

The video annotation data presented in this paper is structured by the underlying AdA Filmontology and was largely created with the open-source annotation software Advene.^[21] Before we delve deeper into the structures of our dataset, we introduce both of these prerequisites in more detail.

The AdA Filmontology, as a further development of the eMAEX analysis routine, systematises analytical statements in a way that renders them machine-readable as well as referring to defined concepts. This comprehensive systematisation of film-analytical annotation vocabulary serves six main purposes: 1) Several annotators can jointly and consistently describe compositional audio-visual patterns. 2) By making the vocabulary machine-readable, annotation data can be enriched with a wealth of metadata. 3) Automatically-generated data (e.g. colour or shot detection) can be integrated into the analysis process. 4) Multilingual annotation work can be merged. 5) Complex audio-visual dynamics can be made visible in a systematised way through the visualisations of the structured data. 6) Systematised film-analytical observations as FAIR research data can be adapted, critiqued, appropriated, and reused by others.

The AdA Filmontology encompasses the structured vocabulary that we use to annotate audio-visual media. The ontology was developed to describe audio-visual compositional patterns through basic film-analytical concepts. Its description layers are independent from each other and could be adopted or dismissed separately according to the needs of future annotation endeavours. The AdA Filmontology is designed as a threefold structure:

Annotation Levels – An annotation level is a category that groups a set of similar annotation types (e.g. all types related to camera or all types related to acoustics).

Annotation Types – An annotation type refers to a concept of the annotation routine under which a movie is analysed (e.g. camera movement speed, or dialogue intensity).

Annotation Values – An annotation value is a concrete characteristic an annotation type can have (e.g. for camera movement speed – slow, medium, fast, alternating).

The ontology allows for free text annotations (in around a quarter of the types, e.g. ‘Dialogue Text’) and annotations with predefined values (in around three-quarters). During the annotation process free text and annotation values are attributed to concrete time spans of a video file. The ontology differentiates between 501 annotation values on 8 annotation levels in 78 annotation types, e.g. the annotation type ‘Field Size‘ distinguishes between eight different values from panorama to extreme close up. Several types and values refer explicitly to established basic concepts of film analysis.^[22]

For the purpose of a framework for corpus analysis, basic concepts that are clearly distinguishable were preferred to more nuanced but ambiguous concepts. The ontology was already adapted and extended by detailed studies and research projects in other subject areas.^[23]

The analytical vocabulary was modelled as a data ontology according to semantic web standards, i.e. the descriptions are machine-readable and can be connected with other semantic data. Each concept (and annotation) receives a Uniform Resource Identifier (URI) that allows us to store and retrieve annotation data in a Resource Description Framework (RDF) database.

Our video annotation data was generated as Linked Open Data through the interplay of Advene (and its capability to export RDF data), the AdA Filmontology (as the data model), and the implementation of W3C Web Annotation Standards and Media Fragments URI.

As Linked Open Data video annotations can be queried and enriched with metadata as well as visual analyses. Metadata enrichment in comparison to classical sequence protocols means that the analytical claim ‘between second 3 and 5 is a tracking shot’ is not stored as plain text but based on the principles of the semantic web^[24] as a node of semantic triples (see Fig. 6).

e.g. the section between 00:00:03 and 00:00:05 of the video file X which is an instance of film The Company Men(the metadata of which can be found in this external film database entries) was classified by annotator Z in regard to the camera work as a ‘tracking shot’ that stands in a certain relation to other possible values of this type.

All film-analytical concepts that can be attributed are human-readable through short bilingual definitions in English and German (see Fig. 7).

The definitional texts are concerned with moderating the tension between the informational architecture of an ontology, which suggests objective knowledge and objects of knowledge with clear relationships, and the humanities’ inherent scepticism toward unquestioned assertions of ontological knowledge.

Via the interface of Advene and a custom template that we publish as part of our dataset, the defined vocabulary can be manually assigned to time segments or (semi-)automatically generated by automatic recognisers. In this way, it is possible to trace basic dynamics of audio-visual staging back to the unfolding audio-visual moving image using frame-accurate and systematised annotations. Film scholars without extensive programming skills can thus generate, visualise, and evaluate semantic data themselves. The implementation of automatic video processing tools was evaluated only in a second step in order not to disproportionately favour values that are easy to quantify in the analysis vocabulary. In addition to the detection of shot boundaries, colour recognisers, optical flow analyses, tone classification, and the detection of trained visual concepts are used.

Advene provided the input interface for our annotation process. Together with Olivier Aubert we integrated the ontology, linked data support, and optimised procedures for large scale video annotation. But in case a project setup does not require Linked Open Data, basically any other annotation software could be used to generate data that is in compliance with the AdA Filmontology (as first trials with the AdA vocabulary in VIAN and ELAN have shown). A timeline view with a video player that allows the precise and multilayered annotation of video files is essential. Initially, we chose Advene as our main annotation platform because of its customisable shortcuts as well as its options for different views and interfaces.

Visualisations based on the annotations have also served as an essential insight-gaining tool in the development of analyses. In order to make the complex (meta-)data readable, our research group, together with Olivier Aubert, developed a dynamic digital visualisation form to display dynamics of cinematic expressivity as graphic patterns:^[25] rising and falling graphs, exemplary stills, and symbol notations come together here to make the basic features of cinematic expressive movement visible in the large number of various different annotations (see Section 6). What is decisive is not the single exact value or the average value of a film, but the temporal dynamics: the speeding up and slowing down, the approaching and distancing, the fading in and away. The timeline generated from annotation data aims to make the dynamics of individual annotation types graphically visible. Detailed instructions for the use of Advene with the AdA Filmontology are laid out in the ‘Manual: Annotating with Advene and the AdA Filmontology’.^[26]

Collaborative annotation practice

The iterative development of the specific AdA Filmontology vocabulary was intertwined with the setup of an annotation process for multiple annotators. This routine was a requirement for the extended annotation and analysis of the corpus that was established in the beginning of the project.^[27] Our team included four student assistants with a film studies background as expert annotators. We met regularly to discuss the vocabulary, the selection of values, and specific annotation choices that were initially tested with selected scenes. In addition to defining our vocabulary and making necessary technical adaptations to the annotation software Advene for the coordinated and distributed annotation of full-length movies, we had to establish intersubjective agreements regarding systematic annotation decisions. These included the thresholds of values or the reference time frame for specific annotation types: what are, for example, the thresholds for qualifying camera movement intensities as ‘minimal’ or ‘static’? Which values are annotated for the duration of a shot and which for smaller or larger units? Should it be possible to assign multiple values within one data point or should that be split into separate annotations?

The relations between annotations are especially important with regard to our analytical interest in the unfolding dynamics of compositional patterns. Relational, ordered values (such as ‘low’, ‘normal/medium’, ‘high’) are context-sensitive and depend on the thresholds that each film or scene defines for itself. This raised the question how to employ these scales for analytical purposes. In cases such as the type ‘Image Brightness’ a workflow was established to archive sample screenshots for specific values within a given video that specify what we consider as dark, medium, bright, or bright-dark in this specific film or broadcast. Another outcome of our regular project meetings was the introduction of syntax elements as part of the AdA Filmontology that allowed for the annotation of evolving (TO) or contrasting (VS) relations between two values (e.g. a zoom-out movement traverses in a fluent transition from ‘closeup,(TO),longshot’). The results of the experiences and concrete agreements gathered during the regular project meetings can be found in the ‘Notes on collaborative annotation with the AdA Filmontology’^[28] that are based on internal protocols of the project-specific annotation process.

We started with the full-length annotation of the feature film The Company Men (John Wells, 2010) across 66 different annotation types which led to a dataset of approximately 22,000 annotations for a single feature film. Due to the time required for a full annotation of several hours per minute of film, not all tracks could be annotated for all relevant films after this initial prototype in the context of this project. Instead, we defined a selection of 22 basic types from all 8 levels that we recommend as a starting point for most case studies. This selection is highlighted in the AdA Filmontology document with a square symbol and it is stored as a timeline view in the Advene template. In the annotation practice additional types can be added if necessary to adequately describe specific examples. We used the reduced subset of 22 basic annotation types for the full-length annotation of the feature film The Big Short (Adam McKay, 2015) as well as the documentaries The Inside Job (Charles Ferguson, 2010) and Capitalism: A Love Story (Michael Moore, 2009), furthermore a selection of features from the German television news broadcast Tagesschau (2008-2009) and the activist videos Occupy Wall Street (Sem Maltsev, 2011) and Where Do We Go From Here? Occupy Wall St. (Ed David, 2011). In the last phase of the project we additionally annotated specifically selected scenes from other films for case studies.^[29]

The annotation routine included an editorial workflow that encompassed various aspects. This involved utilising the checker function within Advene, a feature designed to ensure the accuracy of annotations (such as checking for overlapping or empty annotations, the correspondence of annotated values to the Filmontology). Additionally, an extra keyword, ‘tbd’ (to be discussed), was introduced to mark annotations that required further consideration. The annotation data was consistently revised to align with the most up-to-date version of the AdA Filmontology and the collaborative annotation agreements that were established to facilitate joint efforts and enhance the quality of annotations.

Description of the dataset

Our comprehensive dataset contains more than 92,000 manual and semi-automatic structured annotations authored inAdvene as well as more than 400,000 automatically generated annotations for wider corpus exploration. The annotations are published under the CC BY-SA 3.0 licence and available as Linked Open Data, in the form of RDF triples stored as Turtle files. Furthermore, we provide the Advene packages for the 49 manually-annotated videos in the non-proprietary azp-file format, which allows instant access through the graphical interface of Advene, as well as an AdA template package for annotating video files with the AdA Filmontology in Advene. The template provides the developed semantic vocabulary in the Advene software with ready-to-use annotation tracks and predefined values.

Besides the annotation data we publish the latest public release of the AdA Filmontology itself as an owl-ontology file.^[30] For optimal understanding and utilisation, we offer comprehensive manuals and documentation (in English and German):

• AdA Filmontology – Levels, Types, Values
• Manual: Annotating with Advene and the AdA Filmontology
• Notes on collaborative annotation with the AdA Filmontology

The project’s data is published on Zenodo: https://zenodo.org/record/8328663, (DOI: 10.5281/zenodo.8328663). It can also be found in the project’s public GitHub repository (https://github.com/ProjectAdA/public) and the annotation data can be queried at our public SPARQL Endpoint (http://ada.filmontology.org/sparql).

All annotations can be downloaded, queried, and visualised via the web application AdA Annotation Explorer (https://project1.ada.cinepoetics.org/explorer/) in conjunction with password-protected access to the source video files (Agt-Rickauer, Scherer & Stratil 2022). Its software components (source code and dockerfile) are accessible in the project’s GitHub repository: https://github.com/ProjectAdA/ada-ae/.

To demonstrate the export capabilities of the AdA Annotation Explorer our dataset furthermore contains two different types of file samples: 1) csv files of all manual annotations of the type ‘Field Size’ throughout the corpus, these can be easily imported in most annotation, visualisation, and analysis applications; 2) static html files of customised queries that allow the storage of subsets of our research data independently of the web app’s availability while maintaining the graphical interface.

Interoperability and reusability

As mentioned, the possible scenarios for interoperability and reusability of this dataset are diverse and depend on the intended scales and preferred interfaces. We would like to emphasise two different aspects. First, the ontology can be considered as the main project result for which the concrete annotation data are acting as proof of concept or training data that can be appropriated for different use cases. For this reason, we have included the extended manuals and user reports as central elements beyond the mere function of data documentation. We hope that our exemplary execution of the methodological setup encourages others to apply it to different epistemological constellations, choosing and adding project specific types and values while keeping the principle of Linked Open Data.

Second, the annotation data themselves will be of interest for interdisciplinary research into the Global Financial Crisis, but also for research into the broader field of audio-visual rhetorics as well as experimental aesthetics. Here the dataset offers two complementary pathways: the AdA Annotation Explorer is our main interface for querying the machine-readable Linked Open Data. Queries and comparisons are possible between films and videos via all possible combinations of analytic parameters and displayed as dynamically adaptable visualisations.

All single query results are easily saved and forwarded as html-export for further considerations. At the same time, it is possible to open the single annotation files as azp-packages in Advene in order to add values for previously unaddressed tiers or to export individually customised visualisations for new case studies. This possibility is of course restricted by the premise of access to the specific file instance of the film or video used in the annotation process. During the project phase the coherence of video files between project members was ensured by verifying this via Checksum.^[31] This also points to the constrictions on open research by the different international copyright regulations, unless one is exclusively working with audio-visual material that is in the public domain.

The annotation files in this dataset offer an unprecedented corpus of detailed and structured descriptions of the multimodal, temporal composition of audio-visual data. We can imagine its suitability as a rich set of metadata for the use of the annotated films and videos as stimulus material in empirical psychological and aesthetic research. Finally, another pathway to the dataset from a computational point of view consists in the SPRQL-endpoint for advanced semantic queries of the annotated data formatted as RDF-triples or URIs respectively. This points to a future pool of Linked Open Data that complements data about films with analytical data of aesthetic properties. We are convinced that this mode of describing audio-visual aesthetics makes analytical claims more transparent and allows the microanalytical analysis of larger corpora. We hope that this data framework allows and inspires the reuse of such data for other research questions.

Critical reflections

Reusing the AdA Filmontology and the annotation data of this publication means their re-vision and re-writing for us. Neither the types and values nor the single attribution via an annotation claim universality or being objective truth. They are the results of and heuristic tools for making descriptive claims of perceptual experience concrete and datafiable. They are an invitation to a debate on the tools and competences for a digital humanities approach to film analysis. In addition, the poetics of the films also shaped the episteme of our analyses – meaning that the specific films we were interested in led to certain analysis categories and that different objects of research might need further annotation types and values. The AdA Filmontology concepts are the outcome of permanent trade-offs between detail and specificity on the one hand and generalisation and interoperability on the other, and some categories could benefit from revision by domain experts (e.g. emotion categories). We intensely monitored intercoder reliability in the initial phase. Given the fact that even the border between shots, the distinction between two field sizes, or the distinction of two colour tones are no discrete decisions, aiming for the exact identity of values is not achievable. Our evaluation therefore led us to prioritise a similarity in relational dynamics as the most fruitful goal.

We are acutely aware that simply applying, reproducing, and scaling up the results and the methodology is made difficult by the fact that using the existing data in terms of readability still requires some expert knowledge of the films and that some training and film analytical qualification of annotators will remain necessary. Through the modularity of our working routine, we hope that researchers with different needs and prerequisites can select and adapt those parts that fit their workflow and research interest (e.g. by only adapting some of the categories without engaging in the setup of a Linked Open Data research environment). Implementing a Linked Open Data approach to film studies is based on complex infrastructural prerequisites and the corresponding iterative development processes are time-consuming and resource-intensive compared to a poor media approach to digital humanities.^[32] The question of computational bottlenecks was a reappearing issue since the chosen video annotation software was not designed to handle the combination of our analytical framework’s granularity and the number of annotations. Therefore, Advene (and other tools) had to be constantly adapted for our purposes.

The temporal restrictions of research projects also mean that the annotation data is not complete for all objects of study and that not all discussions of uncertain values were resolved. A statistical analysis of specific levels or values for the corpus would therefore require further data cleaning. Even though the accuracy of automatic classifiers for scholarly analysis was limited during the project phase, we regard the implemented interfaces between concept-led manual analysis and automatic recognisers as a viable proof of concept. Most annotations are still based on time intensive manual labour while low level features from a computational point of view – like cut detection and optical flow analysis – proved more reliable than high level features like object detection.

One of our initial outlooks, to use the project’s annotations for machine learning, did not materialise and this perspective has of course changed rapidly due to the recent developments in neural networks. It remains to be seen how disruptive or productive image-to-text and video-to-text generators will become in general media practices and in research. What semantic web technologies and Linked Open Data can offer in this environment is to open the black boxes of deep learning and to work with categories, relations, and definitions that are both machine and human readable and therefore open to debate as well as to modification. The theoretical benefits of Linked Open Data are many but they require a) resources and willingness to engage with complex data technologies; b) researchers; and c) funding that keeps open-source software alive, as well as a culture of sharing and reusing research data. As a crucial precondition for such a culture, we envision the consolidation of the existing community of video annotation initiatives that, guided by critical media theoretical and epistemological reflection, reaches some viable standardisation not necessarily down to each and every value but on an interoperable system of vocabularies and data structures for film analytical observations.

Authors

Jan-Hendrik Bakels is assistant professor in the film studies department at Freie Universität Berlin and the Cinepoetics – Center for Advanced Film Studies at the same university. Previously, he was principal investigator of the digital humanities project Audio-visual Rhetorics of Affect. He concluded his PhD research with a book on audio-visual rhythms, viewer affects, and film-analytical methods aimed at the empirical reconstruction of audio-visual aesthetics. His research interests include audio-visual poetics, theories on affect and emotion, film-analytical methodologies, digital film studies, and interactive audio-visual media.

Matthias Grotkopp is Assistant Professor for Digital Film Studies at Cinepoetics – Center for Advanced Film Studies and the Seminar for Film Studies at Freie Universität Berlin. He is the author of Cinematic Poetics of Guilt: Audiovisual Accusation as a Mode of Commonality (Berlin-Boston: De Gruyter, 2021). He supervises the project Intervening World Projections: Audiovisuality of Climate Change which is part of the CRC 1512 Intervening Arts, funded by the German Research Council. His research interests include the audio-visuality of climate change and ecological disaster, genre theory, and the relation of politics and poetics, the films of the so-called Berlin School, as well as digital methods of film analysis. He is the managing editor of the open access online journal mediaesthetics based at Cinepoetics.

Thomas J.J. Scherer is research assistant at Cinepoetics – Center for Advanced Film Studies at Freie Universität Berlin. From 2017-2021 he was a member of the junior research group Audio-Visual Rhetorics of Affect. Scherer’s research interests include the aesthetics and poetics of utility films and propaganda, digital research methods in film studies, as well as audio-visual metaphoricity. His publications include Inszenierungen zeitgenössischer Propaganda. Kampagnenfilme im Dienste des Gemeinwohls (Berlin-Boston: De Gruyter 2023) and Cinematic Metaphor in Perspective: Reflections on a Transdisciplinary Framework (co-editor, Greifenstein et al., Berlin-Boston: De Gruyter 2018).

Jasper Stratil is research assistant at Cinepoetics – Center for Advanced Film Studies at Freie Universität Berlin. From 2017-2021 he was a member of the junior research group Audio-Visual Rhetorics of Affect. His research interests include: audio-visual rhetorics, videographic criticism, and methods of digital film analysis. He is author of Audiovisuelle Rhetorik als politische Intervention. Einsprüche der Wahrnehmung vom Kino bis zu YouTube (Berlin-Boston: De Gruyter, forthcoming).

References

Agt-Rickauer, H., Hentschel, C., and Sack, H. ‘Semantic Annotation and Automated Extraction of Audio-Visual Staging Patterns in Large-Scale Empirical Film Studies’, Proceedings of the 14th International Conference on Semantic Systems(SEMANTICS), 2018: https://ceur-ws.org/Vol-2198/paper_109.pdf (accessed on 11 September 2023).

Agt-Rickauer, H., Scherer, T., and Stratil, J. ‘AdA Annotation Explorer: Ein Framework für zeitbasierte Linked Open Data-Annotationen zur audiovisueller Korpora’, Book of Abstracts – 8. Jahrestagung des Verbands »Digital Humanities im deutschsprachigen Raum«, 2022: 30-34; https://doi.org/10.5281/zenodo.6328163.

Agt-Rickauer H. and Hentschel, C. ‘Linked Open Data Management and Multimedia Analysis in the Project “Audio-Visual Rhetorics of Affect”’, presentation at the workshop Corpus Analysis of Time-Based Arts & Media, 6-7 November 2018, Berlin: Freie Universität Berlin.

Aubert, O. and Prié, Y. ‘Advene: active reading through hypervideo’, Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, 2005: 235-244.

Aubert, O., Scherer, T., and Stratil, J. ‘Instrumental Genesis through Interdisciplinary Collaboration – Reflections on the Emergence of a Visualization Framework for Video Annotation Data’, Annual Conference of the European Association for Digital Humanities (EADH), 2021): https://www.olivieraubert.net/talks/20210924-eadh/ (accessed on 11 September 2023).

Arnold, T. and Tilton L. ‘Distant viewing: analyzing large visual corpora’, Digital Scholarship in the Humanities, Vol. 34, No. 1, 2019: i3–i16; https://doi.org/10.1093/llc/fqz013.

_____. ‘Analyzing Audio/Visual Data in the Digital Humanities’ in The Bloomsbury handbook to the digital humanities, edited by J. O’Sullivan. London: Bloomsbury Academic; 2022: 179-188; http://dx.doi.org/10.5040/9781350232143.ch-17.

Bakels, J.H. Audiovisuelle Rhythmen. Filmmusik, Bewegungskomposition und die dynamische Affizierung des Zuschauers. Berlin-Boston: De Gruyter, 2017.

Bakels, J.H., Grotkopp, M., Scherer, T., and Stratil, J. ‘Matching Computational Analysis and Human Experience. Performative Arts and the Digital Humanities’, Digital Humanities Quarterly, Vol. 14, No. 4, 2020: http://www.digitalhumanities.org/dhq/vol/14/4/000496/000496.html (accessed on 11 September 2023).

Baxter, M., Khitrova D., and Tsivian, Y. ‘Exploring cutting structure in film, with applications to the films of D. W. Griffith, Mack Sennett, and Charlie Chaplin’, Digital Scholarship in the Humanities, Vol. 32, No. 1, 2017: 1-16; https://doi.org/10.1093/llc/fqv035.

Berners-Lee, T., Hendler, J., and Lassila, O. ‘The Semantic Web: a new form of Web content that is meaningful to computers will unleash a revolution of new possibilities’, Scientific American, Vol. 284, No. 5, 2001: 34-43.

Biltereyst, D. Meers, P., Verbruggen, C., Moreels, D., Noordegraaf, J., Chambers, S., De Potter, P., Cachet, T., Franck, N., Deroo, F., Mediavilla Aboulaoula, S., Vermeire, E., Waegeman, M., Van Den Berghe, S., Goerlandt, J., Warrens, A., Ducatteeuw, V. ‘Cinema Belgica: Database for Belgian Film History’, www.cinemabelgica.be (accessed on 12 September 2023).

Bordwell, D. and Thompson, K. Film art: An introduction. New York: McGraw-Hill, 2013.

Champion, E. ‘Digital Humanities is Text Heavy, Visualization Light, and Simulation Poor’, Digital Scholarship in the Humanities, Vol. 32, 2016: i25-i32.

Corrigan, T. and White, P. The film experience: An introduction. Boston: Bedford/St. Marin’s, 2012.

Drucker, J. Visualization and interpretation: Humanistic approaches to display. Cambridge-London: MIT Press, 2020.

Eisenstein, S. ‘The filmic fourth dimension’ in Film form: Essays in film theory, edited by J. Leyda. New York-London: A Harvest/HBJ Book, 1977 (orig. in 1929): 64-71.

_____. Towards a theory of montage, edited by M. Glenny and R. Taylor. London: BFI Publishing, 1991.

Ferguson K. ‘Volumetric Cinema’, [in]transition: Journal of Videographic Film and Moving Image Studies, Vol. 2, No. 1, 2015.

Flückiger, B. ‘A Digital Humanities Approach to Film Colors’, The Moving Image: The Journal of the Association of Moving Image Archivists, Vol. 17, No. 2, 2017: 71-94.

Gaines, J., Vatsal, R., and Dall’Asta, M. (eds) Women film pioneers project. New York: Columbia University Libraries: https://wfpp.columbia.edu (accessed on 11 September 2023).

Grotkopp, M. ‘Catastrophe or Pointillism of Disaster? Annotating and Visualizing Patterns of Ecological Imagination’ in Doing digital film history: Concepts, tools, practices, edited by S.M. Dang, T. van der Heijden, and C. Olesen. Berlin-Boston: De Gruyter, 2024 (forthcoming).

Kappelhoff, H. Front lines of community: Hollywood between war and democracy. Berlin-Boston: De Gruyter, 2018.

Kappelhoff, H., Bakels, J.H., Berger, H., Brückner, R., Böhme, D., Chung, H.J., Dang, S.M., Gaertner, D., Greifenstein, S., Gronmaier, D., Grotkopp, M., Haupts, T., Illger, D., Lehmann, H., Lück, M., Pogodda, C., Rolef, N., Rook, S., Rositzka, E., Scherer, T., Schlochtermeier, L., Schmitt, C., Steininger, A., and Tag, S. Empirische Medienästhetik. Datenmatrix Kriegsfilm – eMAEX / Empirical Media Aesthetics. Data Base War Film – eMAEX. 2011–2016. https://www.empirische-medienaesthetik.fu-berlin.de/en/emaex-system/index.html (accessed on 11 September 2023).

Kramer, M. ‘What does a photograph sound like? Digital Image Sonification as Synesthetic Audiovisual Digital Humanities’, Digital Humanities Quarterly, Vol. 15, No. 1, 2021: http://digitalhumanities.org/dhq/vol/15/1/000508/000508.html (accessed on 11 September 2023).

Loist, S. and Samoilova, E. ‘How to capture the festival network: Reflections on the Film Circulation datasets’, NECSUS_European Journal of Media Studies, Vol. 12, No. 1, 2023: 767-818; https://doi.org/10.25969/mediarep/19615.

Melgar Estrada, L. and Koolen, M. ‘Audiovisual media annotation using qualitative data analysis: A comparative analysis’, The Qualitative Report, Vol. 23, No. 13, 2018: 40-60.

Mittell, J. ‘Videographic Criticism as a Digital Humanities Method’ in Debates in the digital humanities 2019, edited by M. Gold and L. Klein. Minneapolis: University of Minnesota Press, 2019: 224-242.

Pfeilschifter, Y. ‘“Inside the Doomsday Machine”. Die affektrhetorische Bedeutung des “Inside” und seine Konsequenzen für das “Outside” im Finanzkrisenfilm THE BIG SHORT’, Mediaesthetics – Journal of Poetics of Audiovisual Images, Vol. 4, 2021: https://doi.org/10.17169/mae.2021.88.

Scherer, T. Inszenierungen zeitgenössischer Propaganda. Kampagnenfilme im Dienste des Gemeinwohls. Berlin-Boston: De Gruyter, 2024; https://doi.org/10.1515/9783111185651.

Scherer, T. and Stratil, J. ‘Can’t Read my Broker Face? – Tracing a Motif and Metaphor of Expert Knowledge Through Audiovisual Images of the Financial Crisis’, Literature Compass, 2024 (forthcoming).

Sobchack, V. The address of the eye: A phenomenology of film experience. Princeton: Princeton University Press, 1992.

Stratil, J. Einspruch der Wahrnehmung: Audiovisuelle Rhetorik und diskursive Selbst-Verortungen der Gegenwart. Berlin-Boston: De Gruyter, 2024 (forthcoming).

_____. ‘Geteilte (Medien-)Erinnerung und die Zeiten der Krise. Zum Diskurs audiovisueller Finanzkrisendarstellungen anhand von ‘Breaking News’, Mediaesthetics – Journal of Poetics of Audiovisual Images, Vol. 4, 2024 (forthcoming).

Tsivian, Y. ‘Cinemetrics, part of the humanities’ cyberinfrastructure’ in Digital tools in media studies: Analysis and research, edited by M. Ross et al. Bielefeld: transcript, 2009: 93-100.

Zorko, R. ‘Bild(schirm)räume der Finanzkrise – INSIDE JOB/CAPITALISM: A LOVE STORY’, Mediaesthetics – Journal of Poetics of Audiovisual Images, Vol. 4, 2021: https://doi.org/10.17169/mae.2021.92.

---

^[1] AdA refers to the German title of the project: Affektrhetoriken des Audiovisuellen. The project has been funded by the German Federal Ministry of Education and Research (BMBF); it is a cooperation between Film Studies at Freie Universität Berlin and the Hasso-Plattner-Institute for Software Engineering in Potsdam.

^[2] Kappelhoff et al. 2011-2016, also see Bakels et al. 2020.

^[3] Sobchack 1992; Eisenstein 1977, 1991; also see Kappelhoff 2018.

^[4] These instructions provide basic procedures for segmentation, description, and qualification as well as text modules and a selection of analytic layers.

^[5] Also see Bakels 2017.

^[6] Agt-Rickauer & Hentschel & Sack 2018.

^[7] Champion 2016; Arnold & Tilton 2022.

^[8] Gaines & Vatsal & Dall’Asta, Women Film Pioneers Project.

^[9] Loist & Samoilova 2023; Biltereyst et al., ‘Cinema Belgica: Database for Belgian Film History’.

^[10] See https://projectarclight.org (accessed on 12 September 2023).

^[11] Tsivian 2009; Baxter & Khitrova & Tsivian 2017.

^[12] Flückiger 2017.

^[13] Ferguson 2015; Mittell 2019; Kramer 2021.

^[14] Arnold & Tilton 2019.

^[15] Melgar Estrada & Koolen 2018.

^[16] https://archive.mpi.nl/tla/elan (accessed on 12 September 2023).

^[17] The tool is defunct. http://www.iri.centrepompidou.fr/outils/lignes-de-temps/?lang=fr_fr (accessed on 12 September 2023).

^[18] https://www.vian.app/vian (accessed on 12 September 2023).

^[19] Looking at the examples published on https://onomy.org (accessed on 30 June 2023) one can see that only a few of the published taxonomies on this platform are actually making use of the RDF-framework, which makes it difficult to map annotations created with different taxonomies onto each other.

^[20] https://tei-c.org and https://music-encoding.org/ (accessed on 10 September 2023).

^[21] Aubert & Prié 2005; http://advene.org/ (accessed on 12 September 2023).

^[22] See for example Bordwell & Thompson 2013; Corrigan & White 2012.

^[23] See for example Grotkopp 2024; Scherer 2023; Stratil 2024.

^[24] Berners-Lee & Handlers & Lassila 2001.

^[25] Aubert & Scherer & Stratil 2021.

^[26] This manual is part of the dataset.

^[27] https://ada.cinepoetics.org/corpus/ (last accessed 12 September 2023).

^[28] These notes are part of the dataset.

^[29] Pfeilschifter 2022; Scherer & Stratil (forthcoming); Stratil (forthcoming); Zorko 2022.

^[30] The vocabulary is also available online in our triplestore: https://ada.cinepoetics.org/resource/2021/05/19/eMAEXannotationMethod.html(accessed on 12 September 2023).

^[31] A Checksum is a small data block derived from a file or dataset, e.g. a video file. This file is stored and linked to the film’s metadata in the database. (Re-)opening an annotation package in Advene and a video file, the software compares the hashed data from the video file with the database to ensure that all annotators are working with the same instance of the film.

^[32] Drucker 2020.