Let’s see what we can see: combining knowledge and perception centred understandings of moving image materiality

Abstract

The changing materiality of moving images and picture sources is a crucial aspect of the space in which screen stories are told. Technologies that capture and present moving images are responsible for our understanding of what we see as audiences; and as makers, how we create reality on screen.

This paper considers a small sample of 1999-2004 texts that regard the materiality and techniques of moving image technologies. There are evident gaps in knowledge within the literature regarding the material status of particularly digital moving images in this period. These are misunderstandings about imaging technologies and technique that have an impact on our knowledge of moving images and how we understand their historic and future materiality. One common thing within the examples discussed here is the presence of arguments only drawing on perceptual understandings of moving images. These claims do not consider imperceptible knowledge-based information that is crucial to understanding the materiality of moving images.

DownloadFull Paper (122kb PDF)

‘All right, let’s see what we can see.’
Gorman to the marines; Aliens (1986) writer/director James Cameron

Introduction

Both practitioners and theorists have much to contribute to the field of the moving image when considering image materiality, and particularly how technology mediates the experience and production of moving images. Technologies that record and present moving images are responsible for our understanding of what we see as audiences and, as makers, how we create reality on screen. This paper considers select theoretical and practical claims from 1999 to 2004 regarding materiality, technologies of the moving image, and the reality the resultant images represent. These claims often consider only perceptual knowledge of moving images and fail to take into account knowledge-based information that is imperceptible, but crucial to understandings, particularly of digital moving images.

There are evident gaps in knowledge within this literature regarding the material status of electronic analogue and digital images. These are misunderstandings about imaging technologies and techniques that have an impact on our grasp of moving images—how they are created, and how we regard those images.

Peirce’s legacy and the moving image

The legacy of nineteenth century philosopher Charles Sanders Peirce ensures that there are numerous references to an indexical break when images are recorded or rendered digitally. These images are most often described in terms of an analogue/digital dichotomy.

At its core, Peirce’s work attempts to systematically describe the relationship between signs and their referents, here between moving images and reality. However historically, and as demonstrated in this paper, the relationship has been leveraged without due consideration of the technological specificity of moving images. In Peircian terms, for a sign to serve as an index, in a similar way to how a book index relates to the main text, a sign must have a direct relation to its object. Peirce (1867) cites several examples of indices, among them the photograph. Within contemporary media theory, there are significant arguments that describe an indexical break in the shift from analogue to digital imaging; that is, a rift in the indexical relation between sign and object—through digitisation (Mulvey 2004, 2006; Rodowick 2007; Manovich 1999, 2001; Balides 2000, 2003 and Nichols, 1991, 1994).

From the early 1980s onwards imaging technology in cinema studies and film theory is most often considered as ‘before’ and ‘after’ the advent of digital technologies. At one end of this continuum, early commentary by Hayward and Wollen (1993) is predominantly hesitant and non-committal in its observation of the new:

‘…the digitisation of the photographic has renewed debates about the (un)reliability of the indexical relationships between the represented and its representation. Digitisation has made the malleability of sounds and images seem like something new and has rekindled anxieties about the ability to communicate truthfully; in an era when communications technologies proliferate, the irony is painful’ (Hayward and Wollen 1993, 7).

Garrett Stewart (2007), at the other end of this spectrum and 14 years later, offers an intense description of indexical severance in carving an analogue/digital divide, appearing to include analogue electronic moving images—not just film—as electro-mechanical technology, against the ‘electronics’ of the digital. He writes:

‘In postfilmic cinema, no image precedes the one we see—or follows from it in sequence. All is determined by internal flux (frame singular). Between whisked-away imprint and the whiplash action of convertible pixels, then, lies the difference between electrical and electronic cinema: segmental transit versus fragmental transformation. Photograms graph motion, inscribe it by succession itself. By contrast, the computed picture, timed by binary (com)mutations, is more like a weightless easel for pixel tessellations, bit by microbit’ (Stewart 2007, 6).

I argue that what is missing from these kinds of arguments is a nuanced view that takes into account successive technologies that are used to create moving images. It is here that practitioners (with knowledge that is not limited to perceptual understandings of the moving image) are well placed to interject in a dialogue that, in this period under study, is dominated by theorists writing about screen production. For example, analogue video uses mathematical sampling to process pro-filmic or a-filmic reality as it is captured on tape or transmitted for live broadcast. Unlike digital processes, mathematical sampling does not quantise the video stream. This sampling of pro-filmic reality is very different to the way in which film technology represents that same reality, yet both are considered analogue technologies; on the same side of a technological and indexical divide. The outcome of this electronic sampling for video may still be continuous, as with film, but in Peircian terms, the resultant image certainly does not have the same relationship to reality as a corresponding film image.

David Rodowick (2007) makes two salient observations. The first is that apprehension of moving images as a means to evaluate materiality is insufficient:

‘…still, the conceptual criteria of perceptual realism, which are restricted to qualities of spatial semblance, are of limited use in helping to understand how photographic and filmic images are distinct, nor do they point the way to uncovering or creating new powers of digital imaging’ (Rodowick 2007, 110).

The second is that these histories, and new ones being written, must reflect the true nature of the technology. Rodowick argues that we need more historical context to understand what a ‘digital ontology’ means, what would make it distinct from an analogue medium, and how it might replace a ‘photographic ontology’ (2007, 96).

Theorising the true nature of imaging technologies through technique

Peircian indexicality is widely regarded as probable for analogue moving images and equally inadmissible when considering the relationship digital moving images have with the reality they represent. I have argued previously that this position is, in part, defensible but considers the technology of moving images with a decidedly cursory approach. Complicit within a knowledge gap regarding the essential status of images in this period, are misunderstandings about imaging technologies.

Veracity of information regarding tools is important when making assessments of any technology. There are many published writings of the period between 1999 and 2004—almost 20 years after the advent of the use of digital technology in motion pictures—that demonstrate commanding expertise related to the methodological application of moving image technologies. But there are also numerous texts that demonstrate a distinct lack in understanding of the technologies represented. Writings by this second group further obscure knowledge of imaging technologies and perpetuate inaccuracies about the role and status of digital production techniques. There are a number of examples of this issue and I briefly examine only four.

Each instance of obscuring knowledge of imaging technologies deals with content specifically focused on the role of moving image technologies through technique and arguments that rely on the material properties of images as crucial to the positions taken by the writers I use as examples.

Digital artefacts and materiality in visual effects

In a 1999 themed issue of the journal Wide Angle, Michele Pierson, an oft-cited writer in the area of science fiction cinema and contemporary technology of the time, makes an aside about the then future of computer-generated imagery (CGI) in cinema and specifically digital artefacts, as the term is used in Visual Effects (VFX). Pierson (1999) writes:

‘The decision to describe certain types of computer-generated images as “digital artifacts” is not one that has been made without reservations. The potential for confusion over this term arises from the fact that within the special effects industry itself, a “digital artifact” refers to an extraneous digital object that that has been marked for removal in the post-production process (eg. The removal of wires used for stunt work). It has nevertheless been retained here because it draws attention to the display of the CGI effect as an aesthetic object’ (Pierson 1999, 46).

Pierson (1999) attempts to make an important distinction in support of an argument treating the CGI image as an aesthetic object but her misunderstanding of the central term’s use in VFX creates far more than an isolated slip. Digital artefacts in a VFX industry context refer explicitly to artefacts introduced after the image has passed through the camera lens. They are created by digital processes involving lossy compression, data corruption via hardware or software failure, or an inherent loss of veracity as part of an image file format. Common examples of digital artefacting are pixelisation, macroblocking, gradient banding, chromatic aberrations (including blooming, haloing or fringing) and aliasing via Joint Photographic Experts Group (JPEG) compression. This definition is the antithesis of the pro-filmic stunt wires that Pierson cites as an example of a digital artefact. The digital artefact is not indexical; it has no real world referent. Digital processes applied to images are responsible for the digital artefact’s creation (Fossati 2009, 287). It is also almost always not an intended part of the rendered CGI image. It is a digital anomaly that, although it could be regarded as an ‘aesthetic object’, is not one in the sense Pierson argues for. My position counters Pierson’s argument about the validity of the CG image and the role she wants to ascribe to CGI. The ‘potential for confusion’ is one created by Pierson, rather than clarified.

The materiality of stop-motion versus performance capture

In 2002 the journal Screen published a paper by Julia Moszkowicz in which she makes an excellent argument regarding the indexicality of computer-mediated films, reminding the reader that there is an auditory indexical relationship between CGI work and reality when CGI characters are ‘voiced’ by actors. Moszkowicz argues that ‘the products of digitality are not totally synthetic, but have the marks of human presence and traces of artistic creativity, inscribed all over them’ (2002, 312). This is part of an overarching argument that animation in all forms can be seen to preserve some semblance of an indexical relationship with reality and is more a set of techniques than a genre. Moszkowicz then makes links between indexical relationships for older analogue forms of animation and the residue of these methods in newer digital processes:

‘This [the tracing of “real” movement] is now achieved using stop-motion animation, a device that allows animators to capture the central points of human movement and figuration (as marked out on a real person or ceramic model), and to translate the information into digital data…. In this respect Final Fantasy [2001] is no exception, despite the fact that it stands at the apex of the digital enterprise. For stop-motion techniques are undeniably at its heart, capturing the movement and gestures of real actors and converting their human expression into digital data’ (Moszkowicz 2002, 312).

In these passages Moszkowicz (2002) is clearly describing digital motion-capture and performance-capture technology, not stop-motion animation. Motion-capture provides an environment that allows digital data-sets to be generated by sensing and tracing spatial movement in real-time. This data is then parsed and applied to digital 3D models as an animation technique. Moszkowicz’s account could be just a misnomer and not impact on an argument linking the two techniques, except that the practices of stop-motion and motion-capture function in diametrically opposed ways.

At the core of stop-motion animation is a production sequence of: prepare a pro-filmic event state; capture that state by recording a single or small number of still frames; change the pro-filmic state; capture again, and so on. Stop-motion capture by name and nature is stop-start and the movement being recorded is anything but—it is necessarily static and is recorded as still images. The motion component of stop-motion animation is the end result of a recorded sequence of images but movement (except for its interval) is not performed or recorded through these still frames. Conversely motion-capture is a data gathering technique that records continuous movement. Although motion capture uses sampling techniques to incrementally capture multiple spatial locations in quick succession, unlike stop-motion animation, it records in real time.

A digital technique that would be more supportive of Moszkowicz’s (2002) argument is rotoscoping. Rotoscoping is the frame-by-frame manipulation of digital motion frames to create a seamless motion sequence. The wire removal that Pierson (1999) describes above is most often achieved through this time consuming process. In manipulating the motion image through individual frames, rotoscoping does not occur in real time. Identically to stop-motion animation however, this results in a seamless motion image. An apparent unfamiliarity with moving image technologies leads Moszkowicz to create an argument that is undone before gaining traction.

Materiality and discrete frames in film and video

Rotoscoping is also central to Ron Burnett’s (2004) discussion of Richard Linklater’s film Waking Life (2001) in How Images Think. Burnett claims that video does not contain frames:

‘By way of contrast, it is quite a shift for a film to be shot in digital video and then transformed into animation. Yet, that is exactly what Richard Linklater has done with his recent film Waking Life. In this instance and as a contrast to Final Fantasy [2001], live video footage was “rotoscoped” or animated, frame by frame. This is not an unusual technique in the cinema, but it is rare for video, in part, because there are no frames in video. Rather, images flow by with little to indicate what separates them’ (Burnett 2004, 219).

Burnett, understandably, argues that video is distinct from film but the basis for this appears to be a misconception that the continuously moving record-head of a tape-based video camera, like the MiniDV cameras that were used for Waking Life, do not record discrete frames. Given this, Burnett also appears to conflate the material state of the recording with the viewing experience. The post-production process of rotoscoping—including the interpolated rotoscoping technique using Bob Sabiston’s Rotoshop software employed on Waking Life—requires discrete video frames to rotoscope. The optical shoot component of the production process utilised interlaced footage from the digital tape format MiniDV, which was then conformed as discrete frames for rotoscoping, with the outcome projected as progressive film. The recording mechanism of MiniDV is notably different from film’s stop-start recording of still frames but despite continuously image scanning, its equivalent to filmic ‘exposure’ is the regular and discrete recording and storing of addressable two-part interlaced image frames.

Materiality and compression in digital video

It is not only theorists that misapprehend moving image technologies. The renowned cinematographer and experimental filmmaker Babette Mangolte (2003) contributed a chapter to an anthology honouring Annette Michelson in which she traces the transition from analogue to digital filmmaking processes from a practitioner’s point of view. Mangolte mourns the loss of film as a production and screening medium. The justification for preferring analogue moving images over digital ones rests on a problematic argument detailing how temporal digital images are created:

‘In the world of digital, time is encoded in a bit-map, and there can be no entropy. In the compression algorithm of a digital image, only what changes in the shot is renewed. That which is the same in the shot stays the same in the digital image, in contrast to the constantly changing emulsion grain from one frame to the next in the film image. The inscription of the decaying body in [Michael Snow’s] Wavelength [1967] is therefore not possible in digital, even in HD DIGI [as used by Lucas for the Star Wars prequel Episode III: Revenge of the Sith (2005)]’ (Mangolte (2003, 264).

Here Mangolte (2003) misrepresents digital image compression, weakening an argument about the superior veracity of filmic images over their digital counterparts. The claim is that only inter-frame compression is used for digital moving images. However digital compression has two broad forms—spatial and temporal. Within these forms, compression of moving image data can be inter-frame (temporal; for pixel differences between frames), intra-frame (spatial; for similar pixels within a frame), or both. Only select consumer-market implementations of digital compression use solely inter-frame compression as this kind of file reduction breaks the integrity of the single frame, thus making it unsuitable for frame-accurate editing but sufficient for playback or rough cutting in camera.

If a frame of digital video relies on other frames in order to be rendered, and one of these frames is trimmed from the sequence it can no longer be displayed intact. When editing with consumer level compressed files, only key-frames that are encoded as intra-frame compressed frames can be addressed as edit points. When applied, industrial digital compression algorithms most often result in lossy intra-frame compression, more efficiently storing pixels of the same, or very similar, hue and saturation values as references to other pixels in the same frame. Even consumer video compression compresses only the chrominance element of the image, leaving the luminance component untouched, as viewers would more readily perceive this loss of image veracity.

Despite Mangolte’s (2003) description of the ‘constantly changing’ nature of successive frames of film as grounds for differentiating between analogue and digital technologies, this description equally applies to all industrial digital moving images that are captured, stored and screened digitally.

Conclusion

Let’s see what we can see—an enquiry that blends knowledge and perception. I argue that between these two, a composite approach plays an essential role in understanding the materiality of moving images. Like early criticism of moving images, we cannot, based solely on perception, presume that what we see is real or is an indexical re-presentation. The shared trait between these four 1999-2004 texts is a misunderstanding as to how imaging technologies work and what this means for the materiality of digital images in particular. These writers are working with predominantly perceptual notions of technology and not engaging with knowledge-based understandings of how moving images are created and experienced.

Despite almost two decades of digital production having occurred when these texts were published, it can be seen that the relative ‘newness’ of digitally mediated moving images presents challenges for those reflecting on moving image production and craft. It is crucial that accurate understandings of media technologies are communicated and discussed so that we harness, employ and explore these technologies—historic and new—more fully. In doing so, we interrogate and examine creative screen work with heightened rigour. Through accurate understandings of media technologies we can promote a fuller dialogue between practitioners and theorists, each offering particular blends of expertise and experience so that what is published as scholarly work representing screen production better represents what it is that screen practitioners do and how we do it.

References

Balides, Connie. 2000. “Virtual Spaces and Incorporative Logics: Contemporary Films as ‘Mass Ornaments’”. MIT Communications Forum. Accessed July 2015 http://web.mit.edu/comm-forum/papers/balides.html

Balides, Connie. 2003. “Immersion in the Virtual Ornament: Contemporary ‘Movie Ride’ Films”. in Rethinking Media Change: The Aesthetics of Transition, edited by David Thorburn and Henry Jenkins, 315-336. Cambridge: The MIT Press.

Brinkmann, Ron. 1999. The Art and Science of Digital Compositing. San Diego: Morgan Kaufmann Academic Press.

Burnett, Ron. 2004. How Images Think. Cambridge: The MIT Press.

Cameron, James. 1986. Aliens [Motion Picture]. United States/United Kingdom: Twentieth Century Fox Film Corporation/Brandywine Productions/SLM Production Group.

Fossati, Giovanna. 2009. From Grain to Pixel: The Archival Life of Film in Transition. Amsterdam: Amsterdam University Press.

Hayward, Philip and Tana Wollen (eds.). 1993. Future Visions: New Technologies of the Screen. London: British Film Institute.

Harley, Ross (ed.). 1993. New Media Technologies. Sydney: Australian Film Television and Radio School.

Linklater, Richard. 2001. Waking Life [Motion Picture]. United States: Fox Searchlight Pictures/Independent Film Channel.

Lucas, George. 2005. Star Wars: Episode III—Revenge of the Sith [Motion Picture]. United States: Lucasfilm.

Mangolte, Babette. 2003. “Afterward: A Matter of Time. Analog Versus Digital, the Perennial Question of Shifting Technology and its Implications for an Experimental Filmmaker’s Odyssey”, in Camera Obscura, Camera Lucida: Essays in Honor of Annette Michelson, edited by Richard Allen and Malcolm Turvey, 261-274. Amsterdam: Amsterdam University Press.

Manovich, Lev. 1999. “What is Digital Cinema?”, in The Digital Dialectic: New Essays on New Media edited by Peter Lunenfeld, 172-197. Cambridge: The MIT Press.

Manovich, Lev. 2001. The Language of New Media. Cambridge: The MIT Press.

Moszkowicz, Julia. 2002. “To Infinity and Beyond: Assessing the Technological Imperative in Computer Animation”. Screen, 43(3) 293-314.

Mulvey, Laura. 2004. “Passing Time: Reflections on Cinema from a New Technological Age”. Screen, 45(2), 142-155. doi:10.1093/screen/45.2.142.

Mulvey, Laura. 2006. Death 24x a Second: Stillness and the Moving Image. London: Reaktion Books.

Nichols, Bill. 1991. Representing Reality: Issues and Concepts in Documentary. Bloomington: Indiana University Press.

Nichols, Bill. 1994. Blurred Boundaries: Questions of Meaning in Contemporary Culture. Bloomington: Indiana University Press.

O’Quinn, Donnie, Matt LeClair, Steve Furth and Tim Plumer. 1996. Digital Prepress Complete. Indianapolis: Hayden Books.

Peirce, Charles Sanders. 1867. “On a New List of Categories”. Proceedings of the American Academy of Arts and Sciences, 7, 287-298.

Pierson, Michele. 1999. “No Longer State-of-the-Art: Crafting a Future for CGI”. Wide Angle, 21(1), 28-47.

Poynton, Charles. 2003. Digital Video and HDTV: Algorithms and Interfaces. San Francisco: Morgan Kaufmann Publishers.

Rodowick, David N. 2007. The Virtual Life of Film. Cambridge: Harvard University Press.

Stewart, Garrett. 2007. Framed Time: Toward a Postfilmic Cinema. Chicago: University of Chicago Press.

Jump to top