Feedback Sep 2010 Production Report

The
following words and images document the September 2010 development of
feedback and associated work-in-progress showings. Much of this
document is written in present tense as feedback is still
under development and will undergo significant change before it
reaches a final incarnation.

I
originally created feedback as an extension of my lighting
design practice. It has evolved into something much more due to the
contributions of my collaborators:

Composers:
Wendy Suiter, Houston Dunleavy and Joshua Craig

Movers:
Jessica Millman, Solomon Thomas, Malcolm Whittaker

This
documentation is intended to, amongst other things, provide practical
insights that might be of use to practitioners attempting similar
projects or who would like to know more about working with the
technology.

What is
feedback trying to achieve?

Feedback
is constructed, conceptually, around the idea of placing humans into
a feedback loop with machines. Both humans and machines are decision
makers within the loop. The long term goal is to produce behaviours
that are both emergent and accessible.

Emergence
is where complex output is created by relatively simple input,
however this is a somewhat subjective phenomenon. For feedback
the term emergence has a number of connotations: it represents an
effective, efficient algorithmic relationship in the sense that it is
more perfectly designed. It also often indicates that the algorithm
is an effective embodiment of the performer, reacting ‘naturally’ to
smaller, somatic movements (when that is an objective of the scene)

Creating
accessible behaviours simply means creating behaviours that hold
interest for viewers. It’s important, if a little uncool, to
acknowledge accessibility when creating intensely technical works. I
unashamedly want feedback to go places in the future and so a
reasonable amount of consideration needs to be given to viewer
experience at all stages of the process. This accessibility is
manifest in the need for scenes to have a dynamic/shape. To go, in
some way, from A to B.

In 2010
the feedback loop functions
as follows: The mover moves onstage, the camera picks up their
movement and sends an image to the computer. The computer interprets
the behaviour as visuals and audio according to algorithms set for
each scene. The mover then reacts to the visuals and audio creating
the next frame of input.

For this
development there are four scenes. Each experiments with different
algorithmic relationships inside the loop. These relationships vary
from linear to abstract. I.E in some scenes it is obvious how the
performer affects the virtual environment, whilst in others it is
more subtle, to the point in scene 4 that it is unclear if they are
having an effect at all.

Where
did feedback
come from?

Feedback
originally comes out of the software I developed during my Masters in
Theatre at the University of Wollongong. At that point I was trying
to create a tool that would unlock the potential of digital
projectors in traditional lighting designs. The idea was to create a
system that allowed you to draw light directly on an actor with a
physical interface rather than having to work with abstract ideas
like channel numbers and parameters. You could precisely light a
face, or just elements of a face, or even draw in false shadows, in
any colour the projector was capable of. The created image would then
stay on the actor wherever they moved within the projected area. It
could be saved so that it was later recalled by a standard lighting
control desk, along with normal lights, so it wouldn’t put any extra
burden on stage managers or lighting operators. I built a working
prototype with two projectors and staged a short performance:
Everlasting Geometry.

I’d used
bits and pieces of the software on a number of projects since, but
knew I wanted more time to explore the new lighting possibilities
offered by this kind of system. I began making enquiries but it was
when I saw Chunky Moves Mortal
Engine that I realised
there could be an audience for this kind of work beyond lighting
designers and other techno types. Mortal
Engine derives some of it’s
appeal as a dance show, but it is arguably a presentation on the
choreography of light. Up till that point I thought light couldn’t
have an audience, it was a show that had an audience and the show
simply needed light. Seeing it motivated me to think big and put
myself in the shoes of a director rather than lighting designer. If I
ever wanted an audience that big to appreciate my work, as I now
believed they could, I’d need to put myself out there.

This new
found artistic impetus presented some questions: As an artist, what
was I offering? Why was light my chosen form of expression? What was
I hoping it could communicate to an audience?

My
answer: When I was starting out I often did the lighting for band
nights in pubs. I only had access to some basic equipment with a few
colours and the interface was usually a manual lighting console –
no memory, and no use for it because you weren’t going to see a
rehearsal anyway. However it was these gigs where I fell in love with
lighting. There is nothing like doing the lights for a band you’ve
never heard before in your life, but suddenly finding a moment where
everything syncs up. You feel the music building and the lights
build. You make the big moments big, the quiet moments beautiful.
It’s like the architecture of this dingy pub comes to life and is
supporting the band, a sum greater than it’s parts. And what I
realised after seeing Mortal
Engine is
that it’s not just me,
everyone recognises it when it all comes together. We feel more than
human because the whole thing, the lights, the sound, the
gesamtkunstwerk,
embody our experience. It creates, briefly, a communal, tribal
connection that everyone can participate in if they choose. Lighting
is, for me, a form of universal communication.

Actually
doing it, having your hands on the buttons, feels like you are part
of a loop, part of the connection between the band and the music and
the audience and the technology. And you don’t have to verbalise what
you are doing, it’s just there without language, you know if the next
moment needs to be big or small or fast or slow but you don’t need
words to describe it. It’s like the you in your head that makes the
decisions is hardwired in there directly and you don’t have to speak
through this big matrix of words and symbols. You just act.

The
experience I want to share with those who view my work is the
experience of being in that loop without spoken language or written
language or even body language. Feedback
is my first attempt to
bring that experience to an audience. And whilst dancers and
projectors may seem a long way from rock and roll I believe it’s a
path that could get there.

What was
the process for the 2010 development?

Feedback
began
with an application to the Merrigong Theatre Development Program.
This was successful and Merrigong generously agreed to provide three
non-consecutive weeks in the Gordon theatre for creative development.
These weeks occurred in July and September, culminating in two
work-in-progress showings on September 8^th
and 9^th
2010.

I put a call out for contributors and
was lucky enough to get three composers and three movers involved.
After some initial meetings feedback was split into four
scenes of 10 – 15 minutes each. The team for each scene was free to
explore the concept of a feedback loop however they wished.

In the
end a wide variety of approaches and techniques were employed, an
ideal scenario for a first development. These included sample based
audio and video, generative audio and video, dynamical systems, AI’s,
arbitrary and evolving relationships.

Scene 1

Visuals/Programming:
Toby Knyvett

Composer:
Wendy Suiter

Mover:
Jessica Millman

Scene 1
ended up having a very ‘organic’ feel, and explored an almost
sentimental algorithmic relationship between mover and machine.

The
sound employed a multi-speaker array arranged in a spiral through the
space. Wendy created four tracks of audio which were routed to the
eight speakers. Several layers of algorithms switched speakers on and
off according to a variation on the Fibonacci sequence. Some algorithms had
priority over others, for instance Jessica spinning onstage would
cause speakers to come on permanently. This gave the whole scene some
shape by ensuring we had more speakers on then off as time went on.
The actual tracks were made from site-specific sound Wendy had
recorded over a period of several months, most often from Thirroul
beach. Elements came and went from the soundtrack over time, adding
to this shape.

Jess
comes from a belly dancing background and the flowing movement style
she brought to the space worked beautifully. Jess also brought in a
veil which at first I was hesitant about. However once we tried it
onstage and I could see how Jess was using it to modify her shape it
became a vital part of the scene. I should take a moment to thank
Jess, Malcolm and Solomon for their patience and energy. I would
often ask them to get onstage and improvise for up to half an hour or
more whilst I pounded out code and rewired things.

There
were two parts to the visuals in scene 1: The first was the ‘dots’,
which is a variation on the original prototype visual I developed and
had been using when showing people what the system could do. The
movement of the dots is based on a mathematical attractor.
Initially I to move away from the dots as I felt I’d already seen all
they could do, however I came back to them because they were just so
damn responsive to Jess’s movements. The texture and movement was
such that over time I found I would see the dots rather than Jess,
I.E see the product of human movement, but not so much the human
body. By reducing the visibility of body language in the piece I felt
brought the viewer closer to seeing the loop itself rather than a
performative arrangement.

There
was a distinct shift where we moved to the second part of the
visuals, the dots disappear and a tree begins to grow from the bottom
of the projection area. The tree shape was based on a Lindenmayer
system and it’s shape was seeded based on Jess’s movements,
which meant no two trees would ever grow the same. In this section we
also see two ‘shadows’ on either side of Jess, these are modified
versions of her silhouette. After the tree has grown to its full
height it begins to break apart, shrinking to become hundreds of
dots, referencing the starting visual. However these dots, instead of
appearing in a grid, are disarrayed, still borrowing their positions
from the tree shape.

The move
from dots to tree, I hate to admit, simply occurred after a preset
time elapsed. For any future versions of this scene I would dearly
love to have this transition controlled by either the performer or
through biofeedback from the audience.

Scene 3

Visuals/Programming:
Toby Knyvett

Composer:
Joshua Craig

Mover:
Solomon Thomas

Scene 3
diverted from the first two in having a non-random decision making
machine. It employed a very simple AI to control an artificial
lifeform.

The goal
of the lifeform was to eat the shape of the mover. It was blind but
could ‘feel’ where the mover was onstage whenever they moved. The
faster they moved the more likely it was to extend a tentacle towards
them. If it could only detect small movements, not enough to give it
a clear reading on where to send a tentacle, it would instead
increase its total size and attempt to catch the mover that way. When
enough of its mass was touching the mover it would devour their
shape. A few moments later it would release but retain a shape
derived from the shape of the movers shadow. In this way the organism
evolved to resemble the mover over time, permanently taking some
amount of DNA from its meals each time it ate.

Audio
was generated in pure data. The same parameters that fed and
distorted the AI shape were sent to the PD patch via MIDI. In this
way the audio and visual expression were very strongly linked, to the
point that the sounds appeared to be the sounds of the organism
rather than the sounds of the mover moving, despite the latter being
closer to the truth.

Everything
the organism did was a result of the movers actions, just like the
algorithms in the other scenes. However the organism is clearly read
as a separate entity as opposed to the dots and tree from scene 1
which were strongly recognised as an extension of the movers body.

Scene 4

Visuals/Programming/Composer:
Toby Knyvett

Mover:
Malcolm Whittaker

Mover
(2^nd
showing only): Solomon Thomas

Scene 4
runs in contrast to the others by purposefully obstructing the
relationship between the mover and the machine response. Technically
it still loops, but not in a way a viewer can easily identify. The
establishing and dissolving of repetitive patterns within the scene
compounds confusion further by providing the viewer with ‘false
starts’ where they believe a relationship is established.

The
vision consisted of 9 viewports displaying Betty
Boop for President,
a black and white film in the public domain. The viewport that
Malcolm stood in front of (and hence was projected on him) was the
only one that ran in real-time, the other 8 views would play any
frame out of the last hundred. The cartoon was divided into clips
between 5 and 20 seconds in lngth. Malcolm’s movement was aggregated
as white noise, which would then drive the system in choosing when to
advance to the next clip and which frame the 8 delayed viewports were
showing.

The
audio algorithm was designed to trigger a drum machine via MIDI.
Every 8 bars a basic drum pattern was randomly generated according to
a Gaussian distribution. By default only a few of these beats would be
played. Malcolm’s movement increased the probability that beats would
be played, until, if he moved fast enough, all drums would play on
all beats.

Malcolm’s
movements explored a kind of non-dance. Instead of any kind of
virtuous movement he would parody movements from the cartoon,
themselves already caricatures of real movement. Most of the time he
would be imitating a movement from a clip that wasn’t anywhere
onscreen so it was not immediately obvious that this was going on.

The
purpose of scene 4, in the work-in-progress showing at least, was to
evaluate the true value of providing ‘supportive’ relationships
between technology and human by taking that support away and playing
them in opposition instead. It also marked a significant departure
from our aesthetic by utilising the visual symbols present in the
cartoon.

Note:
I’ve avoided including scene 4 images until I can 100% confirm
the public domain status of Betty
Boop for President.

What was
learnt from the work in progress showings (and post-show feedback
sessions)?

Overall
there was a positive response. Most people responded well to the
ideas presented within feedback and were keen to see it taken
further.

I think
people found scene 1 to be the most accessible and visually
arresting, many responding well to the dynamic changes within the
scene. Many came away with visual metaphors about fireflies and other
natural phenomenon which I feel is a comment on the expressiveness of
the visuals. Some felt scene 1 best embodied the concept of feedback
in terms of seeing a clear loop, particularly with the audio.

Scene 2
seemed to spark a lot of ideas about the potential of the technology.
Some said they felt the audio wasn’t as responsive as scene 1, which
I believe is to do with the samples being triggered when the centre
of the silhouettes crossed a trigger, rather than their edges.
Because the centre changes as the shape changes it is harder to see a
consistent relationship between shadows and audio. There was also a
comment that one viewer believed there was less of a loop because
Solomon was facing upstage and watching the shadows rather than
facing the audience. In both cases, on a technical level, the reverse
is actually true – in a quantitative sense far more information was
sent that affected sound in scene 2, and arguably Solomon was better
able to participate in the loop by seeing what he was doing. I think
in both cases this points to a certain counter-intuitiveness about
some of these relationships, which I will elaborate further on in a
moment.

Scene 3
was widely recognised as being an interaction between two separate
entities, and the audio was praised for bringing the creature to
life. For me scene 3 is something I want to take much, much further.
Ideally I would like to see a dynamic that would begin with a
response to human expression, aka scene 1, but slowly splitting away
into separate human and machine agency, before perhaps merging again.

Scene 4
provoked a mixed response. Some people found it interesting, if
enigmatic, whilst others found it patronising. Some seemed to feel
that where they had ‘gotten’ what was going on in the other scenes,
scene 4 was cheating them by giving them nothing to ‘get’. The viewer
response to scene 4 illustrated for me the value of the human-machine
relationship as a discrete element within the whole work.

In a
similar way to learning the secret behind a magic trick there seemed
to be a great deal of pleasure derived from seeing how each
relationship worked. But it was never how it really worked, its not
finding out about code or equipment or nuts and bolts. It’s not the
relationship I see as creator and coder. Instead I think the viewer
sees a relationship on an intuitive, cause-and-effect level. And the
actual unfolding of it in front of them delivers some kind of
satisfaction, like solving a puzzle. However the viewer also has
expectations regarding what this relationship should be. There were
many comments about scene 2 and 4 that people ‘wanted’ to see sound
and vision sync up, that it would have been a stronger experience for
them if it had. This is also evidenced by some of the negative
reaction to scene 4, after the expectation was established of certain
relationships in prior scenes that scene 4 failed to deliver on.

So what
might these relationships communicate? This is still an area of
exploration for me, but I’m finding some parallels with lighting
design which is, after all, where this whole process started. I
believe the algorithmic human/machine relationships are political
relationships. They are commentaries on where power rests in the
performative environment. The organic relationship in scene 1 speaks
strongly of ‘natural’ power in the body. Scene 2, with its sampled
shadows, gave power to Solomon as a decision making agent – we see
a picture that Solomon has consciously arranged. Scene 3 also gave
power to a decision maker but began to localise that power in another
entity. Scene 4 took the power away from the human completely, giving
it instead to an arbitrary omnipresent machine (rather than an
isolated entity-machine as in scene 3).

Precisely
shaping the arrangement of apparent power within feedback will
be a vital part of its next development.

In terms
of the viewers themselves I observed a divide between those who were
frustrated by not being able to see the dancer, and those who never
even thought it was an issue. At first I thought this may be about
expectations regarding dance performance and a privileging of the
virtuous body, in the same way written text is often said to be
privileged in dramatic theatre. However I think it also may be to do
with an intuitive rejection of the feedback loop itself, or some
aspect of it. I need to think more on this but certainly it is a
factor to keep in mind for future development.

For me
much of the viewer response resonated quite strongly with my own
feelings on feedback, so one of the most valuable things to
come out of the showings was a validation of my own view of the work.

Questions
I now know the answer to:

Is
feedback a dance work?

No.
Physical movement is a vital form of input in feedback, but I
think the language and traditions of dance are something that would
cloud feedback as a whole work if they were to become a major
element. I do think getting the input of a choreographer is vital for
taking feedback to its next stage. I’d love to bring in an
‘expert mover’ who could help the other participants find a greater
range of physical expression in their interactions.

Is
feedback a theatre work?

No. For
me theatre is distinguished as work that presents ideas in language
and asks us to consider those ideas. What I want, and it’s a big
want, is for language to be largely stripped out of feedback
in an attempt to communicate via…. whatever it is that is left when
you take away language. I can’t describe it well yet. I see it in the
dots from scene 1. The person moves, they extend their arm, but we
don’t see an arm and all the body language associated with an arm. We
just see the dots moving, but we know they are human because it’s
still a human movement.

This is
a difficult decision. The clips in scene 4 are extremely attractive
to me from a visual perspective, and I will certainly revisit them at
some point, but for now they are too loaded with symbols and
language. I think feedback needs to remain true to my original
(and admittedly grand) idea, that there is some kind of euphoric
understanding to be found as a participant in a feedback loop. By
stripping away the distractions of higher, symbol based communication
I believe I can more effectively share my base,

Is
feedback a performance?

No. And
yes. The work-in-progress happened in a theatre and ran for 50
minutes. And I think the version of feedback that I’m
currently making funding applications for will do the same. And when
the next development is over I want to take it to festivals and
venues and show it in a performance format. I think this is simply
because I have a background working on performances and so I
understand how to put work on in these spaces. I think feedback
could also work in an installation format and I’m not ruling out
going that direction in the future. So I think the answer is that its
entirely subjective as to whether feedback is a performance or
not, I only know that its an expansion of my practice and so it hangs
on the same framework as my previous output.

What
does the future hold?

Lots
more of what we just began to look at in scene 3. A decision making
algorithm that can be as expressive as the dynamical systems
(attractors) used in Scene 1.

A
stronger link between the scenes overall, but not going so far as to
start thinking of it as a narrative.

Some way
for the movers to see what the vision is doing onstage without having
to face the back wall.

A high quality video camera to record the documentation.

A
biometric sensor worn by viewers. This signal would mediates the data
coming from the stage allowing the dynamic/intensity of the loop to
be controlled through some intrinsic value coming directly from the
viewer. Not only would this literally put viewers into the loop but
it would allow a ‘natural’ method for shaping the piece, rather than
relying on timed changes or other arbitrary control.

When
feedback is complete I hope to put the entire video of the
2010 September development online in order to complete the
documentation. However in the meantime find some excerpts here

Questions
and responses are always welcome.

I’d like to say a big thankyou to:
Merrigong Theatre Company for supporting my work.
The Faculty of Creative Arts in the University of Wollongong for helping me realise that this was possible.
The vvvv.org community for making this possible.

Photos courtesy of Jessica Millman Photography