080319 – Interface Joi – London

Tools, machines, utensils, apparatus, utilities. There are many sub-divisions and classifications used to describe man’s prosthetics. Their classification relates to how they are used, their level of automation and their scale, they are all tools however, the airplane, the hammer and the factory. Through time, as our tools have changed, so has our means of interacting with them. Once a simple tool, a chisel would take a life time to learn to use with skill. Today our tools have shorter learning curves and greater pre-programmed conclusions of use.

All tools are prosthetics, they extend our reach and leverage our powers. Simple tools such as a club, a knife and a stick are hand held and used to act upon our immediate environment, they hammer, cut and thrust. I would call these first-person tools, that can be used to kill or heal, craft or destroy, farm and feed. They are used to have immediate effect upon an object or subject within our immediate vicinity. They can be used with great skill, to sculpt and paint. The interface with these tools is via touch, they are hand held, tactile, sensorial extensions of ourselves and our thoughts. As these tools never leave the hand and are always monitored by sight, sound and touch, they offer sensorial feedback that allows the user greater control. Modern simple first-person tools offer even greater leverage, for example the electric drill, the electric saw and the pneumatic hammer which greatly increase our power, speed and efficiency but at the loss of sensorial feedback. These tools are controlled by the intellect with feedback generally being of a visual nature. The tools have been mechanised and are products of a mechanised world, their use has an accurate predestined conclusion, to fit a screw, tighten a nut or to accurately cut a material. The tools over empowerment diminishes its feedback but increases its efficiency. Modern first-person tools may be sophisticated and still offer sensorial feedback, tools such as a bicycle, a sail boat and a glider. Feedback is felt and this feeling is part of the tools intuitive use and our interaction with it. Powered equivalents of these tools, the motorbike, the speedboat and the plane still all offer sensorial feedback as the consequences of their power and speed magnifies the forces that act upon them.

Simple tools have also been developed to have effect on objects and subjects beyond our immediate vicinity. Tools such as the spear, the catapult and the bow. These magnify our range as they are thrown or throw projectiles and increase our sphere of influence. These too can be used to kill, hunt and destroy. These, I would call second-person tools. Like the first-person tools these are tactile sensorial extensions of ourselves, their benefit of increased range comes at the cost of diminished accuracy. Their task is usually a simple singular objective, to hit something at a distance beyond our reach. This lack of control and limitation of the tools use means that they are rarely used to heal, sculpt or paint. Modern versions of these tools for example the rifle and the harpoon offer even more power and accuracy over ever increasing distances. The feedback from their use has been intellectualised, it is cognitive, pre-destined, and is predominantly controlled by sight.

On many modern tools the component we interact with is not the tool itself but is remote from the tool, connected via wire, blue tooth, satellite, the web or the cloud. This has numerous considerable consequences on our relationship with the tool and with its powers. These I would call third-person tools, and they have had and are having profound effects on the latter part of the twentieth century and the beginning of the twenty-first century. Third-person tools have enabled the vast accumulation of power and capital within the hands of a few. Unlike the first-person and second-person tools, third-person tools have further controls over distance, scale, boundary, multiples of, and can be self-correcting and algorithmic.

The tool operated by a remote is first and foremost impersonal and objective. There is a distance between the remote and the tool, maybe covering a few centimetres, a few meters or thousands of kilometres. The remote may operate a tool in the yard or on the moon, the distance has little conceptual relevance. The operator may be sat by a pool on a beach operating a tool in a war zone or at the depths of the ocean operating a tool in the desert, and although this is unlikely, the emphasis on detachment and consequence is important. With distance one can be coldly impersonal and objective. All of the sensorial feedback from a first-person tool is lost on a third-person tool. Control over huge distances enables both the spread of power and the concentration and centralisation of power. The third-person tool operates in a variable distance environment.

The remote also has no bias to scale, it may operate the infinitely small or the infinitely large, trans-oceanic tunnelling machines, satellites in space or nano-surgical tools. With the remote one could open dams or bridges, heart valves or arteries. The third-person tool operates in a scale-less environment.

The remote may operate tools beyond a physical barrier or within an inhospitable zone, the tool can operate beyond boundaries and is highly permeable. For example, this may be from behind the safety glass dividing two rooms or from the inside to the outside of the human body. It can operate either side of a physical barrier but it can also operate within zones that are otherwise inaccessible. These may be toxic, radioactive, atmospheric. It can operate tools within the vacuum of space. The third-person tool operates in an environment without boundary. 

Just as it can navigate distance, scale and boundary it can also be multiplied. In commercial terms this would be called scaling or roll out. If one can operate a hammer with a remote at a distance, one can equally operate one hundred hammers, set in format, with the same single remote. A hundred shovels digging, a thousand lathes milling. The third-person tool operates simultaneously in an environment of multiples.

Third-person tools may be algorithmic and have some built in autonomy. Why operate a single drone by remote when you can operate many flying in formation. If each individual drone has a certain amount of autonomy, collectively it could hive mind and swarm. Each drone may have a specific task or bias but still work within the swarm. The hive mind would confer and adapt to conditions ‘on the ground’, as the environment changes the response changes. Collectively it adapts. The third-person tool could operate instantaneously in a changing environment.

The most sophisticated tools that the majority of us use will be our computers, pads and phones. These are all remote third person-tools. We interact with them directly to produce a digital outcome that exists within a virtual digital world. We may use them to type or to photograph, we can draw and sculpt within this digital world, we can operate drones or our homes connected systems. In each case our instructions are digital and are then reprocessed to re-enter the physical world via a remote tool. The printer, the 3d printer, the drone, the thermostat for example. Some of this work may forever stay digital and virtual but still requires an additional tool to be useful and realized. Text, film and image all require a screen, sound requires speakers. These may be proximate, part of your phone, pad or computer but they could just as easily be at a distance, scale-less, multiplied and recurring. These are the elements that make these modern tools so powerful.

We interact with these tools via typing, touchpads or touchscreens using taps, swipes, expansions, double clicks, haptic click, rotations. Typing via writing devices is already three hundred years old. With typewriters being invented in the 1700’s, the first patent in 1714 by Henry Mill. The QUERTY keyboard is also over 250 years old, patented by Christopher Sholes in 1868. Yet our other means of interacting with these machines tapping, swiping, brushing are even more primitive. We can control via voice but language is proving difficult for machines to interpret and they often make errors. The above tools need to be physically touched to operate. 

Touch control can be distanced, using a touchless user interface. This is very similar from mouse to computer interface. Nintendo uses a Wii-remote, a hand held device that communicates positions in space with the computer using motion sensor technology. These were first used in the video games industry in 1972. In the film Minority Report (2002) when agent John Anderson (Tom Cruise) communicates with several screens the hand held device is replaced by interactive gloves. To communicate with the machine agent Anderson uses gesture and voice control. Wired gloves use magnetic or inertia tracking. Some interactive gloves have fibre optics sewn into them that carry light pulses. When the fibre optic is bent, i.e. bending the fingers, the fibre optic leaks light registering a loss and the losses can be tracked. These kinetic user devices require the communication between two tools, one hand held or worn and the other tracking movements.

In the film Her (2014), Theodore (Joaquim Phoenix) plays an interactive game where his hand movement is mapped in real time, alongside using voice control. Gesture control maps movement in 3D space time. To achieve this mapping, it uses depth aware cameras that use structured light to structure depth from a known source via deformations that occur as the light strikes a known surface. Structured light is used in 3d scanners, but more commonly used in multi lensed cameras that use time of flight calculations to resolve distance between camera and the subject. Gesture control uses algorithmic mapping to follow physical movement and to interact with a computer. The interaction is usually visualised on screen. Gesture control using algorithmic mapping is already quite sophisticated and used in facial recognition and in real time links between the face, facial avatars and emoji. Complicated surface modellers like NURBS and polygon meshes can be tracked, mapped, translated and mutated. Distance creates a latency between the tracked subject and executed data. Technology is quickly addressing this, GPS can real time track our phones our watches and our fitness bands. It is possible to tell if the user is walking, running, cycling or swimming and measure distance travelled, average speeds whilst collecting health related data.

Facial motion capture electronically converts the 3D map of a person’s face into digital data. This is now often used as security access for pads. This data base can then be used to produce CG computer animated avatars in real time e.g. facial expression emoji or Apples Animoji. If we can map our faces and have the expressions translated in real time to Animoji it is only a few short steps to animating our expression on to those of a digital other. The digital other may be a president, a celebrity or your pet cat. If we can create a three-dimensional digital version of someone or something we can, via 3D printing, create a real three-dimensional static version. A robotic version controlled via gesture control and algorithmic mapping would not stretch our imaginations, it is only our technical inability that at present prevents this. Gesture control could in theory be used on any non-automated robot, container cranes, underwater diggers, asteroid mining equipment. These would be third-person tools, without distance, boundary, scale or multiple.

Man is known as the tool using animal. If no specific tools are available man instinctively improvises and adapts. He will use clubs, levers, ties and grips, anything that is available. The Industrial Revolution, brought repetition, mass production, uniformity, which increased speed and efficiency, through the scale of the factory. It mechanised our farming and our wars and our relationship to our tools became distanced. The highly personal use of the sculptor’s chisel, the artist’s brush and the musician’s violin bow would become an historical footnote. First-person tools have been demoted from a primary to a secondary field of influence and then further demoted to a realm subservient to the machine, to fix and repair. With the Third Machine Age, The Electronic Age, the interface between man and his tools has been further stretched. First-person tools that were solitary, unaccompanied, tactile and intimate have been removed and replaced by Third-person tools that have a collective detachment. Our personal control has been diminished and fragmentised into a world of pre-organised tasks. A world in which we usually contribute only to a fractional proportion of the total task. 

What is the future means of interface between man and computer and man and AI?

15.30mins

In Blade Runner 2049 the relationship played out between a hologram Joi (Ana de Armas) and the replicant K (Ryan Gosling) explores how one may interact with a holographic companion and be able to sense their physical presence. The scene opens as replicant K enters his flat and begins a conversation with someone else as yet unseen, one is led to assume it is his girlfriend in another room. 

As K arrives home he hits a button on a wall console to his left to turn on his home entertainment system. Frank Sinatra begins to sing ‘Summer Wind’ and a female voice Joi feigns surprise at not hearing K arrive and proclaiming that he is early. Joi tells K to get cleaned up and begins a conversation about his day. When Joi is asked about her day, she says that she is ‘getting cabin fever’, as if she is a stay at home housewife. After his shower K goes to the kitchen and cooks an instant meal, poured from a packet into a saucepan. K asks Joi if she wants a drink, she agrees, K pours two drinks and takes them to the living room along with his bowl of instant noodles. K drinks both drinks. K does all of the tactile work, he eventually sits down and prepares to eat his instant meal. Joi as yet unseen proclaims that she has been working on a new recipe and that she will be with him soon. Joi says, ‘I should have marinated it longer, I hope it isn’t dry’ a real-world physical thing. Joi also gives some background information to the Sinatra track playing, as only a computer could. Joi further exaggerates a real time delay as she is putting the finishing touches to his meal. Joi has yet to arrive and the audience is unsure who she is but Joi makes a big thing out of getting ready for K and preparing his meal. 

Joi opens as a hologram just in front of the kitchen door. She at first appears translucent but quickly becomes more-opaque. The mechanical ceiling mounted projection beam aligns itself as though Joi is coming from the kitchen into the living room. At this point in the film Joi has never left the living room, her space is dictated by the realm of her projection mechanism. K’s actual meal is a bland bowl of noodles and protein nutrients. As Joi materialises, she is carrying her own holographic meal, an aesthetic meal, a hologram she has concocted that she then sits over the real meal. An aesthetic enhancement, visual spice, although there was very little to marinate in steak chips and salad, it beats the look of a bowl of protein slime. The concept of touch is reinforced as Joi the hologram bends forward to kiss K on the cheek, again here the tactile is only visual and would need to be mapped in real time perfectly in 3D space time. A virtual friend will always leave one feeling alone as we have a deep-down need for touch and sensorial companionship. Physical contact is constantly simulated to enhance the reality of the event and intimacy of friendship. Joi lights K’s cigarette with her finger, one assumes, like the holographic meal overlaid onto a real bowl of protein noodles, the lighting and the burning is holographic, a 3D image on the end of an unlit cigarette, complete with holographic smoke. Joi blinks and instantly changes outfits, a pink skirt and white top become black trousers and black crop top, at the same time she changes her auburn hair. Joi reaches for a real book on the side table, Vladimir Nabokov’s Pale Fire, a book about life, death and afterlife. Joi takes a virtual copy of the book and asks K if he wants to read. Joi then changes outfits again, a silver sparkling mini dress, now with blond hair, she asks K if he wants to dance. Joi’s role as a companion is to entertain K.

K has a surprise for Joi, a present, an Emanator. K connects the Emanator to the home entertainment system transferring Joi’s data to the Emanator. Joi disappears. K then turns off the mechanical ceiling mounted holographic projector, opens the hand held Emanator and Joi re-appears. Joi now in a blue dress with long dark hair walks under the shutdown ceiling mounted mechanism and feigns disbelief. Joi spins and giggles as K tells her she can go anywhere she wants in the world now. She has holographic freedom, freedom to roam, her augmented world now extends beyond the living room. K asks Joi where she wants to go first and they walk out onto the balcony to stand in the rain. Joi’s previous world, the confined space of the living room could be easily choreographed, every object within the room had already been copied to Joi’s data base. Joi’s augmented self could seamlessly interact with this environment. Now Joi has become portable, she is no longer confined to the overhead mechanical projection beam, but this new freedom comes at a cost. 

Joi’s new landscape is unmapped and has to be established in real-time, her AI has to quickly assimilate and interpret her new surroundings. To emphasise this transition as Joi steps into the rain it at first passes straight through her and she does not get wet. The concept of a hologram getting wet is the same as a hologram casting a shadow, it needs to be able to achieve both to be grounded within the real space. As the rain hits the holographic Joi it is shown as an electronic haze, a confusion, at first there is little reaction other than perplexity. Joi’s data banks search for a generic rain and the consequences of being in rain and so Joi then begins to get wet, raindrops fall on her hand at first, they are blue. Eventually her hair and clothes get wet and rain droplets accumulate on her skin. Soon she is as soaked as anyone left standing in the rain would be, she is holographically drenched. She expresses the body language of being wet, she has the expressions of being wet, she is inundated, flooded, her skin sodden, dripping holographic drips. Joi moves towards K and they try to emulate touch, K wiping holographic rain from Joi’s holographic cheek while standing in the shower of real rain. They caress, there is longing, desire, affection and tenderness, then suddenly Joi freezes hit by an incoming voice message from K’s employer, this triggers the automated Joi system override and K’s illusion of emotional intimacy is lost. K takes the Emanator out of his pocket and Joi his personal companion is turned off, vanished at the flick of a switch. Joi contained in the Emanator then travels in K’s pocket.

The idea of the Hologram overlaid onto subject, static as meal, moving, later in the film as the surrogate lover. If you had a holographic girlfriend, she could be overlaid onto your holographic real friends, male or female, or onto any subject, animal or mineral. If you had a holographic avatar you could wear this persona over yourself, over your friend, or duplicated over numerous friends, whole clubs full of people like you. Why would you have just one holographic avatar when you could have several, male or female, animal or alien. The accessorised peacock becomes digitally accessorised. Why would a holographic avatar stay as it is when it could morph, grow larger breasts, a muscled chest and abs, hair could visually grow, skin could change colour or surface texture, your face could be visually beautified or aged. The coming augmented world offers infinite possibilities.

The beauty of the Blade Runner 2049 scene within this augmented world is the interface between the virtual and the real, a step nearer to a seamless oneness, when virtual and real are so intertwined that neither can be distinguished. The main computer interface has moved a long way from the keyboard, long gone are the days of typing green code onto a black screen. Gone too are the mechanical rituals of swipe, tap, double tap, expand and contract. Gone too is the interface of two-way instruction, Siri what is this? can you find that? can you book this? questions and answers, the beginnings of elementary conversation but not a conversation as such. In these scenes the A.I. is as near human as it can be, Joi has gestures, mannerisms, confusions and frustrations, she smiles, laughs and cries, she makes impulsive suggestions, she leads and follows a conversation. Joi has opinions but these are never confrontational as Joi also has a clear objective, to be a complimentary partner, an A.I. companion and friend. Joi is an A.I. enhanced companion, a software programme that adapts to and learns from the needs of its owner. Joi can read moods and adapt responses. There is an intimacy between K and Joi that would suggest they have been together for a considerable time and that during that time Joi has processed an in-depth profile of her owner and her owner’s needs.

At present we mainly use our hands to interact with our digital media. We would use our hands to operate the remote that in turn operates a tool at a distance. Yet fighter pilots armed with technological visors already target and fire missiles using their eyes.

36mins

When we are first introduced to Niander Wallace, the CEO of the Wallace corporation (played by Jared Leto), his role is God like, an isolated messiah, part cyborg, an enhanced human. He is about to inspect a new model of replicant. He has been trying to design a female replicant that can reproduce so that the replicants can speed up the process of colonising the off-world planets. His assistant replicant Luv (played by Sylvia Hoeks) brings Niander a set of attachments through which he can operate remote objects. The box for the attachments is triangular in cross section a pyramid extruded with a sliding top. These are sacred technological objects. Luv removes one accessory and places it just behind Niander’s ear, on the mastoid bone and a light in Niander’s neck lights up to confirm contact. Doctor’s already implant titanium into the mastoid bone, this stands out through the skin and is used for connection to external hearing devices. Niander uses this attachment to control six small drones through which he can see. One is led to assume that the drone enables considerably more than sight and can analyse the internal workings and structure of the new model replicant. The drones fly around the replicant in 3D space, controlled by Niander’s brain transmitting information to and from the drones. The over emphasis on the triangular section wooden box (wood no longer exists in Blade Runner 2049 as all trees are now dead) and the gadgets within is to visualise the film so the audience can understand how Niander communicates with the drones. Today Bluetooth connects to any equipped device, we have no need for a range of dongles each with a separate use. If the brain could communicate directly to objects the remote tool becomes an immediate extension of thought. Both the art work of Stellac and work with the severely disabled have explored the beginnings of thought control. 

54mins

Joi has access to a huge data base archive, as she recalls the holographic wooden horse logged from K’s memories. The hologram to be further ‘grounded’ has sound, as Joi flips the wooden horse it slaps into her hand, as Joi puts her arms around K her plastic jacket crackles as it creases. Joi passes back and forth though K but there is a constant realistic sound to her impossible movements.

60mins

In this scene K has been shot down on his way to the orphanage in ruined San Diego. K is outnumbered and being attacked. Luv has been watching him, tracking him via a drone, she sees that he needs help. She brings the drone within range to help K. Luv is still at the Wallace Corporation, she is sitting cross legged in a 66 white Pierre Pauline Ribbon Chair having her nails laser tattooed. Luv controls the drone via voice control and via her glasses where she can focus on a target and a blink initiates the drone to fire. The scene puts emphasis upon the detachment between the remote control and the event. The remote is a fashion item, beautiful glasses. Luv is being pampered, sitting in her designer chair within composed surroundings fighting a war against an army in a scrapyard in a contaminated radioactive zone. Luv is presented as exquisite and feminine, she is fighting a marauding, heavily armed gang without stress or emotive expression. It requires as much concentration as she simultaneously uses to get her nails laser tattooed. The dissociation, remote to event is given emphasis in every way. It should be remembered that Luv is a semi-autonomous replicant programmed to serve Niander Wallace and as such we have a semi-autonomous remote controlling a remote.

1-22mins

Joi, the A.I. hologram is aware that she cannot fulfil K’s physical needs as she lacks a tactile dimension. Joi pays Mariette, a replicant female to become a surrogate lover for K. K had met Mariette previously and Joi knew that he liked her. Joi syncs her holographic body over Mariette’s, the result is an ambiguous combination of the pair. K can now touch Joi via the surrogate’s body. To help visualise what is happening the sync of Joi and Mariette is allowed to lapse revealing the surrogate within the holographic shroud. Joi would need to create and map,’ a NURBS or polygon surface model of Mariette, mapped in real time with any delay being too slight for the human eye to register. 

1-55mins

Joi and the emanator are both products of the Wallace Corporation. When Luv fights and defeats K the emanator falls to the ground and Luv knows the consequences of its destruction. Joi appears and begs Luv not to destroy it as her very existence as compiled data from her relationship with K exists solely within it. Joi is an accumulation of memory that has a three-dimensional interactive presence via the hologram. If Luv destroys the emanator the accumulated data that is Joi is also destroyed. The essence of the person is destroyed. In this short scene Luv is very aware that she is killing Joi in the same event as killing a real person. Although Joi can be repurchased as the hologram as she is a bought product, the essence of Joi and her relationship with K has been built up over years of companionship. In the film her entire being exists within the emanator, without a backed-up memory. In this scene Joi was not cloud linked as that link could be traced by her makers, the Wallace Corporation. In the usual circumstance Joi would need to be cloud linked (or some future equivalent) to be able to interact in real-time with her mapped surroundings. If Joi was cloud linked, as would normally be the case, she would be constantly backed-up in the same way that iTunes or other apps are constantly backed up, accumulating each new alteration or modification.

If the essence of a person is the accumulated memory of one’s experiences, then one would need to question the value of real over unreal experience as in memory they are all equal. One should also note that memory is selective and that the process of selective editing composes a fiction. However, if memory could be digitised and stored it can equally be modified or altered. This will be the dilemma of both cloning and cryogenics. All those presently frozen in liquid nitrogen waiting a new body will have to accept whoever’s memories they are given on trust when the technology to deliver their new body eventually arrives.

Joi and K exist in an augmented reality where the real and virtual overlap. The emanator is the remote purchased and possessed by K and the third-person tool is Joi. The tool’s task here is to be a companion of which intimacy is integral to the role. Usually the benefit of a third-person tool in which the remote controls the tool is that of distance, scale, barrier and multiples as described above. Here, all of those vanish to accentuate the reality of the experience with a companion. The collected augmented experience becomes K’s reality and it exists and is authenticated in the real world in real time. Fiction and fact are inseparable. As at 1-47mins as K asks “Is that real” (dog)? Deckard (Harrison Ford) replies “I don’t know ask him”. The point being does it matter?

In the film The Matrix (1999) when at lunch, aboard the Nebuchadnezzar, the teams ‘digital pimp’ Mouse (Matt Doran) offers Neo (Keanu Reeves) time with ‘the girl in the red dress’, Mouse’s digital creation. In this event all reality is stripped from the interface of the remote as it simply becomes a transition, a means by which the real is replaced by a digital reality. The interface is not a seamless transition from the real to the virtual, but instead the experience would be an either or, real or digital. However, the memory would be real and it would be collected and assembled with other memories to become the essence of one’s being. 

Fictive worlds have been part of man’s creative conscious though-out time, they are often used as a means of escape, as a means of control, or as a means of explaining what one does not comprehend. Fictive worlds exist within and feedback to real world events, from the Greek adventures with the Gods, religions and their associated parables, myths of nymphs, pixies, fairies, zombies, orks, chimera and super heroes. Since the Sumerian gods Enlil, Enki and Ninhursanga (3000BC) to Astro Boy, Witchblade and Dokkoida (21^st Century Manga) the fictive world continues to influence us as they feed back into our real world. Watch a child at play, embrace their imagined fictive world whilst running around the play-ground of real space time. These fictive worlds should never be considered to be dissociated from our real world as we have fought wars, built temples, organised cities, mapped skies, created calendars and other measurements of time, mass and distance around their fictive existence. Man organises and assembles his real world in the shape of his fictive world. The human construct sits on nature and is rarely part of it. It is ordered and organised in a way that is completely abstract to nature. The directed myth controls the real. We do not need to physically touch something to influence, to have causal affect? Joi is an intimate associate, a confident, a sounding board, a guiding hand, a conscience. An AI collection of personal data that is corporately owned. At present we have influencers, they guide the masses to the markets and have great power and persuasion. The idea that we each, at some point in the future, may have a holographic friend, a corporately owned, personally targeted holographic conscience, is frightening. As a future controller of people, it would be almost god like, beyond objective criticism as it would be a collection of self-monitoring personal data. However, it would be under corporate control and act as a collective global conscience, guiding each individual with AI driven personally targeted directives. Here the fictive myth, personified by one’s holographic friend, would have causal and directive effect. Controlling the myth controls and directs the real-world consequence, as it has done through-out time.

Man’s interface with the world via the semi-autonomous third-person tool will allow him to become so distanced from reality and the consequences of reality, that both the real and fictive will become a seamless whole. Our hands already perform a bizarre choreographed ballet on our phone screens as we communicate with distant others, each of us at our gateway to the net and this digital world. Wii game and augmented reality have already made that dance a figurative abstract. The dissociation of movement from anything within its immediate environment will be one of the strangest ballets yet… or is it. The mime artist, the shaman, an emotive performance in classical ballet all connect the real with an imagined world. The real-world movement of the performer is dissociated from their virtual imagined worlds that they describe. This abstract dance, the interface of the future, will operate machinery in underwater mines, build structures on distant planets and walk the inside of our arteries, cleaning and repairing their surface from toxic impurities. Not unlike the performance of the Houngans or Mambos that connect to the afterlife of the Voodoo world, a possessed medium at a séance, or a partygoer on hallucinogens.

Finally, the filming of the holographic scenes in Blade Runner 2049 were not as straight forward as they initially seem. At first it is tempting to believe that they are a simple 2D overlay onto a 3D film sequence, the overlay having transparency and carried out post production. A 2D overlay would not work as it holds no volume and cannot accommodate rotation. As an actor rotates through 360 degrees a 2D overlay cannot correct perspective. To achieve the 3D hologram of Joi, Ana de Armas’s body was mapped and a digital model made of it. This is a 3D surface model, a shell. This was then cut in half vertically, from head to foot so that there were two half shells, a back shell and a front shell. The two half shells could be given different levels of transparency as required to achieve effect. Ana de Amas would act her role and be point mapped in space (usually neck and hips). The 3D model of Joi could then be overlaid onto Ana de Amas as she walked through 3D space. This process to overlay Joi would be the same for Mariette in the surrogate lover scene. Transparency and sync were manipulated to give the scenes visual comprehension.

The relevance of the mapped 3D surface model is important with regard to the future of tool interface. In the film, the 3D holographic shell is the conclusion, it is the objective of the exercise, a visual thing. Holograms are usually used for visual observation. In Blade Runner 2049 at 1-12mins we meet Deckard’s daughter at work in her lab creating artificial memories. She carries a tool that controls her holographic world. We first see her in a holographic forest studying an insect, she later uses the same tool to create a birthday party. The hologram is a 3D rendering of her ideas, of her design, it is a visualisation. But this technology can be reverse engineered. By mapping the body in real-time we could interact with distant spaces and operate distance machines. The software used to create the effect of the holograms and digital modelling in films such as these will soon filter into the augmented reality apps that we will all be using in the near future. These in turn will develop the technologies required for real-time distance interaction using figurative choreography with real world causal machines. The dilemma is that the more distant we are from the consequences of our actions the more fictive those consequences become.

Images

1. Joi’s entrance (15mins)

2. K’s food, Visual Spice (17mins)

3. K’s cigarette (18mins)

4. Data base merge (54mins)

5. Surrogate lover (1-22mins)

6. Single hand sync (1-23mins)

7. Billboard Joi (2-11mins)