Ascendance: an experiment in colour music for VR

Sightlines: Filmmaking in the Academy Issue 3 2021

Louise Harvey, Mark Douglas Williams, Peter Moyes: Audio-visual concepts
Louise Harvey: Graphics and animation
Mark Douglas Williams: Music and 3D audio spatialisation
Peter Moyes: Producer
Film: Ascendance: an experiment in colour music for VR
Length: 6.56 minutes
Year: 2020

RESEARCH STATEMENT

This research project is a collaboration between composers of music and vision for 360 cinematic VR. Colour or Visual Music boasts a strong tradition in the field of abstract animation including luminaries such as Oskar Fischinger, Jordan Belson, Norman McLaren, Mary Ellen Bute, John and James Whitney, etc. Often these practitioners worked on pre-recorded music in applying form, colour and movement to an interpretation of sonic scores. Sometimes the audio-visual synergy was the result of genuine collaboration between animator and musician. The uniqueness of the Ascendance project lies in the challenges the VR canvas presented in its seemingly limitless spatial affordances, and in the genuine cross-disciplinary collaboration required to effectively meet these demands. An original musical score was revisited for re-composing in VR space in anticipation of its animated visual expression, and in consideration of the viewer’s experience. The musical composer and animator worked together to consider how musical aspects such as pitch, volume and tempo might inform both audio and visual staging, and the visual aspects of colour, form and movement.

Initial consideration of audio-visual correspondences was broad, taking in the proposals of, among others, visual artist and theorist Wassily Kandinsky, psychologist Robert Plutchik and the writer Goethe. Published schemas were then put aside as the three contributors —artist, musician, producer— noted their personal responses to the musical score in terms of colour, shape, size and dynamics. Correlations between us were identified, and some basic tenets concerning volume size, pitch-hue reconfirmed.

CGI artist and animator Louise Harvey set to work on crafting colours, settings, lighting, etc; each draft was reviewed by composer Mark Douglas Williams, Louise, and Peter Moyes as creative producer, towards complementary and impactful music-animation relations.

A key finding of the research project is the necessity of a flexible workflow/approach that acknowledges the equal status of visual and audio elements (and contributors), embraces an iterative back and forth dynamic in testing and refining audio-visual relations, and features a nimble attitude in responding to the constraints and liberties of the VR canvas discovered only through research as practice.

It was found that the specific dynamics of sound and vision in the VR space necessitate a conciliatory approach. When mapping vision to sound, the narrow(er) field of attention for the staging of visuals in the VR space can be at odds with the 360° immersive canvas enjoyed by the sound designer. Vision occupies a distinct location in space, while sound can be broad and amorphous. Subsequently, for our project, a visual interpretation of the music focused on key audio motifs and ‘hero’ instruments, and oft-times a reading of the broad emotional sweep of the music, in order to avoid too many visual elements competing for the attention of the viewer across the wide VR space. The music then took its cues from animation as the original 3D sound mix was remapped to echo these visual stylings. Ultimately, the objective of engagement was deemed to be emotionally-driven cohesion and flow over strict audio-visual correspondence.

The project was insightful as proof of concept testing the efficacy of technical/procedural and creative methods, and importantly, ways of working collaboratively across disciplines towards evocative immersive VR experiences in colour and music.

 

PEER REVIEW 1

I really enjoyed the immersive experience of the work, and it became clear to me that combining audio and abstract animation can be a very powerful tool to arouse emotion and a sense of the sublime. I felt it worked best when there was no conscious attempt to force me to look at a specific object and I would argue (actually, I have argued this in my own VR research!) that trying to direct the viewer’s gaze is a hangover from traditional filmmaking: VR is about finding ways to provide an experience wherever the viewer is looking – music can be a spectacular help in that regard.

For me, the two most significant/interesting parts of the research are (a) the freedoms that the VR environment provides audio-visual creators (not really dealt with in the Research Statement) and (b) the decisions that get made connecting audio material (score) to visual material (animation shape, movement and colour). Either of these would make for a fascinating contribution to knowledge.

I think the work is really engaging but not properly aligned with a research statement, which is trying to cover too many bases. I’d suggest some tightening of the research statement to either better align it with the opening comments of the VR work, or possibly remove these comments and let the work point to one of the other research areas raised in the current research statement.

I say this because the work starts with quite detailed remarks about Belson & Jacobs Vortex Concerts, with a concluding comment: “imagine the freedoms Belson & Jacobs could explore in dynamic 360° audio-visual space.” This situates the film work quite specifically and suggests a project examining creative freedom (presumably from physical constraints like projection and sound equipment). I felt that the Research Statement was not entirely aligned with this driving suggestion – referencing Belson (but not Jacobs nor the Vortex Concerts).

The research statement touches on both areas of collaboration (complex workflow and creative decision-making by artists from different decisions) and the problem of how to deal with a loss of control of the frame in a VR environment. Whilst there is clearly innovation and insight, the specific question for the research is not really very clear (and differs from the focus suggested at the start of the film work).

 

PEER REVIEW 2

Ascendance is a very enjoyable experience which showcases an excellent collaboration between animation and composition to create an aesthetically exciting video project. Overall, I would greatly like to see the project submitted to many international film festivals in order to determine the community’s response to it, and to how it fares with other contemporary 360-degree video works. I applaud the use of 360-degree space which produces mostly a comfortable viewing experience, and for the most part an enjoyably novel experience of turning one’s head in synch with the various musical accents that work in concert with the various eye-drawing aspects of the animation. The spherical environments are pleasingly created, with a good sense of colour and aesthetics. It is easy to imagine this piece presented in a gallery setting or perhaps at a music conference/festival event.

Despite the enjoyment of the experience, the video work does not live up to its potential. As a 2020 output, the project engages with some still-relevant aspects of 360-degree video production. But nevertheless, as a piece of research, I would expect that there would be some reflection on the work’s impact. (Certainly, the ERA review process requires this.) I viewed the video multiple times, both in an Oculus Go headset and with the Quest link via an HTC VIVE. There is no difference between the 360-degree video version and the Oculus Quest link, so it is a little odd that the Quest link is provided.

The music is the real highlight of the video, followed by the animation which, more than anything else, shows an excellent control of particle systems and a delightfully creative touch in their use. However, as presented, I am unconvinced by the use of “VR” (or really 360-degree video) as a central element to the project. Indeed, the piece opens with text describing Belson’s use of immersive installations with “strobes” and a variety of other projectors in order to create an experience that would work with images “free of space.” The video text then promises that Ascendance will show the kind of experience Belson may have created with the affordances of VR. Against this background, it is unfortunate that the video is monoscopic rather than stereoscopic, which severely limits the play with “space” that is afforded by VR. As such, it is unclear what the work does that is different to (or better than) Belson’s highly experiential and immersive installation work. Belson’s projections meant the work needed to find illusory ways of having images “fall down” onto the viewer. 360-degree video, in its stereoscopic form, should enable a greater capacity to generate this illusion. Or at least that would be a reasonable research hypothesis. Perhaps a more convincing research statement would say that Ascendance is a proof-of-concept of how Belson’s approach could be made more portable, and thus more easily showcased to audiences without the requirement of a major installation. Such an argument would be convincing and persuasive of its creative research component, and more than that it would also showcase a true potential for VR as an exhibition medium. I do note that the statement suggests this is a proof-of-concept piece, but a proof of what concept?

If considered as a proof-of-concept, which the research statement implies, then I am much more encouraged by the work. However, the research statement does not clearly state or contextualise the work in relevant fields, or in its truly innovative experiment. I am unconvinced by the degree of innovation applied to the “multidisciplinary” argument made by the research statement; animators (or traditional filmmakers) and composers work together all the time. Much more relevant would be a more detailed discussion of the iterative process of each of the artists grappling with the affordances (and extreme limitations) of the 360-degree environment. For example, what methods were used to determine whether or not the spatialisation of particular musical sections should be synchronised with the accompanying animation elements? Did the authors consider/trial having the spatialised music cue begin half a second earlier than the associated visual element, in order to see how that affected the user experience? Or perhaps half-a-second is too much, or not enough, or perhaps the artists settled on having them synchronised precisely at the same moment because they found (via some form of testing) that this was the most effective. Reflection and/or evidence of such process would truly make this a novel research outcome.

Without stating it, the work does present new knowledge in the form of demonstrating a portable form of immersive art installations. I think this is a genuinely valid use of 360-degree video, which potentially offers ways for a reconsideration of whether or not “installation” pieces can be successfully recreated and redistributed via the portable 360-degree format. But in terms of practice as research; does the work need to directly acknowledge what it truly contributes to the field? Or is it enough for the work to simply contribute something and a reader draws it out? Perhaps this is a matter of debate, but I feel strongly that to justify something creative as research there needs to be a clear research question that is worked through with an acknowledged methodology and direct statement of the resulting outcomes.

In light of this, the research statement indicates some interesting areas where it could be developed further. For example, there is an account that the artists revisited psychology and art theory, which was then “put aside” so that the artists could explore their personal responses to the music. I am highly in favour of phenomenological approaches, but why is this not stated? Why was the truly interdisciplinary opportunity of applying art and psychology to the project simply “put aside”? Why was the personal response deemed more relevant? And how was personal response articulated and how did it inform the creative choices made? The statement also indicates that there was a particular workflow practice that emerged during the collaboration process. This could contribute to future experimental work in the field: what was this process? Did each artist have a VR headset that they viewed draft works on and made comments? Or did each artist find some way of visualising what the 360-degree outcome would be without using that? It’s unclear, and these are real questions that would provide valuable insight into the process, practice and research of the work.

There is no doubt that the music is excellent, and it is likely that some solid reflection and accounting of the practice undertaken by all artists interacting would have justified the creative work as a research output. For example, how does the spatial sound mixing of the music interact with the composing process, and how is this different (or continuous with) other forms of experimental binaural audio recording and mixing of musical experiences? Overall, it seems to me that this is an interesting creative work, and which is highly enjoyable as an audiovisual experience in a VR headset, but I am unconvinced it contributes to the research as practice debate. I would have liked to be convinced because I think the field desperately needs more work that demonstrates how VR screen production can be a form of research. I also see indications of where this could have been undertaken. Perhaps instead of presenting this as a non-traditional research output, it would be easier to argue for its place as research by generating a traditional research journal article that reflects on the practice.

 

RESEARCHER RESPONSE

Firstly, we are appreciative of the high levels of interest, energy and critique with which the reviewers engaged with our project – thank you. To respond to the questions and issues raised across both reviews, we provide the following explanations.

The reference to the Jordan Belson, Henry Jacobs 1957-60 Vortex Concerts at the Morrison Planetarium, San Francisco that preface the VR work, is primarily to acknowledge the work of these pioneers and to locate our digital project within a longer tradition of immersive colour music, its roots of course in analogue methods. The fancy we pose: ‘imagine the freedoms Belson and Jacobs could explore in dynamic 360° audio-visual space!’ serves as a segue from analogue to digital, and is meant to establish the notion of our being ‘spoilt for choice’ regards the digital affordances of space, access and iteration. To engage in a thorough comparative analysis of analogue v digital methodologies (and effects) is beyond the scope of this project; rather, we chose to focus on the procedural challenges confronting us in working across disciplines in the VR space.

The opening comments do, however, on reflection, encapsulate our somewhat naïve assumptions in embarking on this project — the 360 canvas and the ability to work digitally (speed, access, flexibility, iterations) we had assumed would open up tremendous compositional possibilities. And yet the breadth of the space presented us with too many options — as creators, we craved the limitation of the traditional frame(!), while, in consideration of an audience, we were sensitive to locate our visual theatrics within the relatively limited visual field of the human eye. The finding that within a limitless space, this visual range was at odds with the broader more amorphous scope of human hearing, necessitated specific ways of working between visual and musical elements and between respective composers. It is these processes that are the key research contribution of this project and summarised in the research statement as:

the necessity of a flexible workflow/approach that acknowledges the equal status of visual and audio elements (and contributors), embraces an iterative back and forth dynamic in testing and refining audio-visual relations and features a nimble attitude in responding to the constraints and liberties of the VR canvas.

Further below are some of the specifics of this workflow that could not be accommodated within the 500-word research statement.

We are happy to flag that Ascendance has been awarded Best New Media at Women In Film and Television (WIFT) Australia 2021 for Louise Harvey as Director, while hopefully, further festival screenings will also evidence research impact. Plans are afoot to stage Ascendance in the traditional format with a live musical performance of the score in September 2021 at the Queensland Conservatorium —this will provide some evidence towards any future assessment of traditional v. virtual presentations.

Regarding the process of deciding on visual-aural correspondences for Ascendance, a phenomenological approach was also beyond our scope; our recounting the referencing of traditional schemas by the likes of Kandinsky, Goethe and Plutchik, and subsequent confirmation of some of the more generally-held correspondences (such as big = loud, lighter colour = higher pitch, etc) with our personal responses, was simply to report our particular approach, and not a study in itself.

For Ascendance, we worked with a finished musical score, adapted/reworked spatially for its VR composition; for future projects, we would like to work from the ground up across both audio and visual aspects, to iterate musical development into the unfolding of the animation itself for a wholly collaborative VR colour music.

Further details on workflow, aesthetic and technical considerations:

Decisions regarding sound spatialisation were the result of an iterative approach, where changes were made to both visual and aural elements in an effort to produce the best viewing experience and to capitalise on the 360 space. Changes were made to the animation’s timing, location and number of animated objects, while the location of audio was accordingly reworked and finessed to create as close a sound correspondence to the visual objects as possible.

While a limiting factor in sound spatialisation was the ‘virtual’ ambisonic software’s reliance on HTRF based algorithms in “simulating” complete spherical spatial sound perception — notably front/back and up/down perceptual ambiguities — we also wanted to explore how much direct visual correspondence could offset this limitation and help to increase the perception of sound location and motion in these directions. Future versions should improve these simulation-real world correlations, including the likely development of individualised HRTF data for each person based on head size etc.

The viewing experience was a major influence in the creative process and provided us with a number of parameters to work within. For example, we acknowledged the fact that many people can experience nausea and dizziness from the VR experience, and we attempted to minimise this response by keeping the camera in a static position. This was actually one of the factors that led us to the decision to opt for a monoscopic output, as the stereoscopic effect is primarily noticeable when the camera is moving, where the parallax effect becomes more obvious.

Nausea can also be induced if the viewer moves their head and/or body quickly and frequently, and this became apparent in some of our early animation experiments where the animated objects moved very quickly through the 360 space. In order to keep a fast-moving object in view, especially when there were many of them, the viewer found it necessary to move around excessively. In response to this, we reduced the speed of movement of separate objects.

Similarly, decisions regarding the size of the moving objects needed to be made in consideration of the disparity of the objects’ apparent size in the editing software as compared to their apparent size when viewing in a Head-Mounted Display.

Dr Louise Harvey, Mark Douglas Williams and Dr Peter Moyes.

Return to list of worksNext work

Jump to top