Clear methodological approach is fundamental to every research. It is also the hallmark of effective artistic experimentation. During my research connected with the documentary, I mainly used inductive, bottom-up analysis approach to search patterns which could establish the visual style and scope of CGI extension added to the picture. I started with series of detailed tests and worked out the solutions by examining related issues. There are two arguments justifying my choice of reasoning. When I began cooperating with the director Pawel Ziemilski in January 2016, he had already worked on the film for three years. During this process, several possible visual solutions or stylizations were initialized, but none of them sufficiently fitted the story. Usually, in this situation, all the answers which were not eliminated during the development seem equally good or bad. In feature films, a production VFX supervisor is responsible for such kind of selectivity, as well as for realizing the director’s vision. In many other instances, unfortunately, such a professional is involved only during pre-production. A documentary is not usually associated with using special effects, especially if initially only some undefined form of a visual stylization is planned. If a pre-production process is meant to be more secure, it should include someone who can suggest adequate technical solutions. At present, it is almost certain that most film realizations will require some kind of post-production. This means that a creative technologist should take part in the production of a documentary from the very beginning.
The second reason for employing the inductive analysis was the lack of a decisive storytelling dominant. The director was not sure which of the themes of the film should be the leading one. The story had too many threads and developed in too many directions. Such multiplicity makes it difficult to establish the criteria for selection. Additionally, individual stories of particular film characters were similar and had a limited dramaturgical potential. People presented in the film were not professional actors or naturally gifted in acting. Unless their story was not accompanied by traumatic experience, they were usually focused solely on their everyday life, and this did not constitute a solid material for creating the plot. Therefore, the method of working from the specific to the general seemed the best possible choice.
The first challenge was to develop the capturing system for Skype. The director had the protagonists’ permission to record their conversations, which were initially the main component of the film. Ziemilski wanted to explore the experience of presence connected with the sense of community sustained only by Skype chats. Thanks to video capturing, it was possible to follow the fate of both those who decided to leave for Iceland and those who chose to stay in Poland. Recorded Skype conversations were also a valuable information resource, tracking any recent changes in the lives of the people depicted by the documentary. This Big Brother-like social experiment was fundamental for the film. My artistic research at that time focused only on the mainstream VFX methodology, so I thought that the strategy of acquiring materials is not really related to my area of interest. Nonetheless, I decided to devise the system of Skype video capturing to establish good cooperation with the rest of the team.
Along with the evolution of my academic interests, my understanding of the research topics has changed. Thanks to my artistic research, I now prefer to describe my position as that of a creative technologist. My analysis led me to the conclusion that the change in the scope or area of activity of a VFX supervisor demands that we acknowledge the transformation of its role and its name. The abovementioned development of the Skype capturing system was a perfect example of such a transformation. I designed the solutions to acquire video materials and the original, stylized manner of their presentation. The visuals captured in this way are of a low quality. Footage is highly compressed, it has a lot of noise, and the recording cameras are low-resolution with a poor low-light performance. Moreover, each participant used different devices from variable manufacturers: smartphones, tablets, PCs, and Macs. Because the important factor for each artwork is visual integrity, having to deal diverse low-quality footage presented us with two possible solutions. The first one was to improve the quality, which may be costly and time-consuming, the second was to level down the quality and introduce additional stylization to cover imperfections. The degradation method was better because some of the materials were planned to be acquired with professional cameras and then stylized to resemble Skype conversation. It was impossible to achieve that degree of quality with the original Skype recorded footage. I found several visual tricks to add relevant distortion and interference to disguise the difference in quality. The director accepted the result, but the visual side of recordings did not fit the final artistic concept, and only the sound part of the conversations was used in the finished film.
During the realization of my first task, I noticed the unusual phenomenon of resistance of the film crew against getting beyond a particular level of technological advancement. The producer, Lukasz Dlugolecki, initially planned to install recording software on each computer separately, then capture visuals without any control, and finally copy the recorded materials from hard-drives after three months. With more than fifty participants, such a system was not very convenient. Instead, I proposed a number of solutions, including a cloud-based system where all conversations could be recorded globally on the server. Although Microsoft Poland approved our project and they were able to offer us an affordable solution, this idea was abandoned and a less advanced method, based on remote computer access, was applied. We tested this technique for several weeks before the final implementation. In my view, the abovementioned limit of acceptance for technology is only natural. A creative technologist or a technophile does not have such a restraint: every new technology is challenging and fun. For most of the population, however, it may pose a problem. The users may feel lost and insecure. In the same way, a conventional artist can accept a only restricted technological leap in the creative process. For instance, a significant number of artists employ a method of manual conceptual drawing on paper and then the digital edition of scanned pictures. Computer is a transparent tool of early creation only for some of them. People prefer traditional methods supporting the thought process. This technological barrier was particularly evident later when the production moved forward.
In addition to the Skype sessions, the film initially was supposed to involve several distinct techniques: animation, scripted static shots, and hand-held “organic” observational scenes. They were supposed to interlace with each other. Each of these elements was meant to describe a different aspect of the society portrayed in the film. I was asked to develop several possible solutions for these components and to explore the visual possibilities. The main topic was the etiological myth which started the emigration trend. To distinguish the past and the present, the director decided to use animation.
The ships sailing under the Icelandic flag reach the Polish port in 1980. An Icelandic navigator Valdi decide to travel across an alien land in search of a wife. The first person that attracted Valdi's attention in Poland is the train conductress Joasia. He falls in love with her at first sight. Despite the language barrier, he manages to determine that she is single and lives in Stare Juchy. Without hesitation, he decides to visit this distant place in the Masurian forests. This adventure will lead to the migration of one-third of the population, four hundred people, from this town to Iceland.
When I joined the team, they already worked on the idea of single-cell animation superimposed over live footage, with the voiceover of interviews with founding fathers. The director asked me to explore other stylistic possibilities. My first idea was to use a map theme and a journey motif. Valdi was a navigator, so the next logical step was to use stylization to 1980s computer graphic or ASCII animation. The narrative could be based on letters from Valdi to Joasia. I also tried to use the method of communication between ships as an inspiration. It referred to a type of work Valdi did.
The second idea was to test South Park style animation. I tried out the stylization combining Polish Masurian folk and Icelandic folk. The plan was to use the combination of drawing and cutouts. Ziemilski wanted the story to progress with a seamless transition between a cartoonish look and a photographic quality of Skype. The main problem was how to gradually change stylized animation into real photographic footage. Unfortunately, the results were not fully satisfactory. The connection of the love story with a funny cartoonish look did not seem to work for the film. This kind of absurd humor was not adequate to the mood that the director intended to get. It emphasized the weakness of using the Skype recordings, because they seemed rather trivial. Finally, the founding story of Valdi and Joasia was abandoned after this research. This allowed me to clarify the definition of digital visual design and to check out the potential of a creative technologist.
Animation is not included in the VFX activity but may be added to it as a separate technique. This possibility comes from the fact that the European film industry does not have such a strict assignment of responsibilities as the Hollywood industry. In Norway, many cinematographers perform the tasks of an on-set VFX supervisor themselves. Computer generated animated films apply many visual effects techniques, but they are a different kind of creativity. In visual effects, computer-created objects, characters, or environments serve to support live-action story and are usually are intended to look photorealistic even if they depict fictitious situations. [1] While VFX supervisors and digital visual designers create elements that are incorporated into live action shots, creative technologists are not limited in such a way. The digital visual design students of the Norwegian film were frequently asked to resort to animation, and this technique is sometimes the most efficient solution for issue various issues encounters in feature film production. Nevertheless, the animation was outside the scope of my artistic research, and In Touch gave up this direction of development.
Complicated relations between members of different families were the core of the story. The network of relationships between the characters initiated for each of them a sequence of personal experiences. The director’s idea was to create a gigantic family tree and present its fragments to show the connections between people. At the bottom of the tree was Joasia, the Founding Mother, at the top was the youngest generation of emigrants. I used a system of mind mapping to establish clear and editable links between the research objectives. This part of development allowed us to construct the story better. Using paper-based notes was impractical, so I employed cloud-based Google Docs as a universal collaboration tool. To represent graphically the form of the family tree, I suggested stylizing 3D rendering to microscope photography. The result was supposed to be realistic, similar to medical visualizations. Even though it was a computer-generated animation, because of its realism it could be classified as a visual effect. After some consideration we abandoned this idea, as it did not match the desired storytelling style.
The next task in the development was to introduce the context of the colonization of an undiscovered land. This context was supposed to be our exposition, the background information necessary for the viewer to understand what is going on in the story. [2] I put forward a number of suggestions, including the use of a tilt-shift effect. In traditional photography, this effect is created by mechanical rotation of the focal plane through specially constructed lenses. Visually it gives an impression of scale change; people look like ants and buildings resemble scaled maquettes. Familiar sights look like video transmission from an unexplored planet. There are many artists, like Bjorn Vermeersch, [3] who highly specialize in this type of photography. Unfortunately, the tilt-shift equipment can be inconvenient to use on the set, especially in a documentary realization. It requires meticulous setting, which takes a lot of time, while outdoor documentary filming demands rapid action and fast decisions. In Iceland, everything is weather permitting, and it is hard to catch the moment when one has to use uncomfortable equipment. This solution requires planning ahead of time. However, the effects are easy to replicate in post-production and are affordable for every budget. The computer made tilt-shift effect is based on a gradual blur. In more complicated cases the amount of blur is mapped on the greyscale bitmap. To give the impression of a real lens, visual artifacts are often added: chromatic aberration, vignetting or an additional directional blur. Animating and setting the effect is easy. The best solution is to connect both digital and analog technique to get a rich and multi-layered result. Usually, this kind of stylization is used only for external landscapes. It challenged me, so I did some test on the potential of using such effect indoor and possibilities for digital enhancement. I looked at it as a challenge, so I tested if this effect could be used indoors and if digital enhancement of the effect was possible. My experiments confirmed the theory that it is most effective for outdoor photography. This effect was easy to replicate digitally, but building interiors never achieved such special scale as landscapes could. The stylization could be done, but the lack of space changed the context in which the effect would be perceived. Initially, the tilt-shifts used in the film were to be mixed alternately with other elements, in a final edit were grouped together as the beginning of the movie, and lastly, only some of them were collected in one place, in the middle of the film. Initially, the tilt-shifts were supposed to be mixed alternately with other elements. In the final edit they were grouped together as the beginning of the movie, but the final/theatrical cut/version used only some of them, collected in the middle of the film. The director decided that they fit emotionally the story of only one character, since it is told in an oneiric form of an unreal dream. In the background, we hear the voice of a person from Poland, with a strong echo effect added. This person is imagining Iceland.
Since we had ruled out several options, the most dominant elements were “postcards”, consisting of two parts. The first one showed a person against the landscape, with the off-screen voice telling his or her story. Because of the limitations of the vocal performance of characters appearing in the documentary, the director planned to write down interviews and engage professional actor for reading. At a later stage, we wanted to make it with tilt-shift, but eventually, this component was replaced by classical shots.
The second part of the “postcard” presented staged scenes where the protagonists were talking to the camera. The scenes used a vertical or horizontal screen division: one part of the screen would show a person, the other would show the subject area of his/her talk. Originally the scenes narrated short stories about the protagonists’ experience in Iceland. Different “postcards” had different styles, according to the profile of a particular character. Because of the form, they looked like personalized messages from the most remote part of the world sent home, to the village, to the relatives, and to the film audience. These one-shot static scenes were supposed to describe how Iceland influenced people’s lives, how it enriched them and their families. The subjects were asked to describe what they have achieved and learned, how they have changed. The scenes were intended to show how exotic Iceland is for newcomers and to explain what they added to the community. One example was a man naming all Icelandic fish he caught and showing them at the bottom part of the screen.
Instead of dividing the screen I suggested using in-frame composition based on multiple layers. I made a test scene with a moving car and presented it to the director. I believe that it was the breaking point for our development. Afterwards, we decided to use the film means in an unusual way. The new goal was to show the connection between the people similar to a Skype chat, but by using film effects. We came to the conclusion that the characters of the documentary pretend to be together, trying to act as if they had never left their home country, but it is obviously not true. In fact, they miss contact with their close ones. Instead of showing direct conversations, we decided to use the prerecorded footage to depict a different emotional bond and deeper feelings. Instead of showing Icelandic geysers and icebergs, the film could present what actually is important for the characters. The “postcards” could show who characters actually are; they could reveal their individuality. I found several methods to make multilayer frame compositions. The easiest one was to use on-set compositing with an LCD screen or a projector, but I wanted to try something more sophisticated. I suggested using VFX in a creative way to combine different pictures in one edit. Instead of dividing the screen into two parts it was to be divided into two planes through a composition based on green screen technology. We decided to try this idea first as the most challenging realization option.
In October 2016 we went to Iceland. It was my first encounter with the locations and the protagonists. An effective development requires getting to the source of inspiration. I wanted to test my ideas on real locations. While I was there, I made extensive documentation of the places, people, letters, and mementos from the past. It was an excellent source for further development and also, as it turned out, the material for the ending of the film.
In order to have virtual access to these locations, I made several high-dynamic-range panoramas and photogrammetries. These actions are commonly used in average VFX realizations to share with the post-production team the information about the location that was available to the shooting team. Having such information, you can measure dimensions, design the set, examine the placement for the equipment or make the next location scouting in virtual reality. During our journey, we encountered enormous, beautiful northern lights. I recorded them using time-lapse photography. Some of these materials were used later for the production. Documentation at such a level is often treated as a final source.
360° In Touch VR experience 01360° In Touch VR experience 02360° In Touch VR experience 03360° In Touch VR experience 04
My next challenge was to find the way to involve the protagonists to a greater extent during shooting. My idea was to explain to them our intentions, to tell them how we want to create the film and how we plan to use special effects. The film was intended to bring together, in its own way, the members of a community separated by thousands of kilometers. We wanted to reunite families and friends through VFX realization, and they had to understand that. After a short explanation, they transformed from passive interview subjects into the members of our team. Another positive aspect of this strategy was that they started to bring their own creative solutions for the scenes. It was possible because these days special effects became the contemporary standard. Everyone knows how they work and what their purpose is. Advanced technical explanation was not necessary. The characters just needed to understand basic VFX rules and the main idea of the film. We tested this method a couple of times. In the beginning, I was required on-set to explain it in simple words, but later the director started doing it himself. Murch claims that the human imagination is able to recognize ideas more powerfully than we can articulate them. [4] You always understand more of the language than just verbal communication. People presented in the film required planting an idea of the movie in their heads. It allowed them to be able to reveal their stories better. Before, the material was more humorous than poetic. Now we had a proper building material to create the story.
Scenes enriched with VFX effects explained how Iceland influenced people’s lives. This created the connection between those who are left behind and those who migrated, but pretended that they never left and they are together with their loved ones. The VFX-enriched scenes showed the evolution of the individuals, the families, and the whole society in the context of values such as freedom, openness towards the world, the opportunities given by rapid mobility. Modern technologies, like mobile phones or Skype, changed people’s knowledge of themselves and of the ones around them, as well as the understanding of controlling their ways of life. Instead of shooting Icelandic geysers and icebergs, the film could present what was really important for the characters.
Every work of art is based on contrasts. You cannot have loud without silent or bright without dark. Each element is relevant to the other in the process of creating a composition. To develop the story we needed to present the influence of the Icelandic culture on emigrants by means of comparing them to the people who stayed in Poland. The first layer of the composition was filmed in Iceland and was based on the experience connected with the country. This part functioned as a background, the environment where we would add the person from Poland who was also involved in the story. We planned to film the second component in Poland with a green screen background, to allow easy post-production. In addition, it was congruent with psychologically based dramaturgy. The feelings in the people who were left behind are much more intense than in those who migrated and who live two lives. The light from the first component was supposed to be recreated on the second set. Such a strategy demanded full technical documentation from the Islandic sets with the high-dynamic-range panorama included, which delivers light calculation information, indispensable in such a restoration.
We planned to use a whole spectrum of special effects tools, including photogrammetry, tracking, green screen, and deep data. We did not want to make VFX completely invisible, to emphasize the technological impact in the picture, and to maintain the impression of documentary truth. We needed to optimize resources to avoid blowing up our limited budget. In addition to proper stylization, every VFX “postcard” scene had to be properly planned to avoid unnecessary additional work. Everything can be corrected in post-production, but it may tremendously increase expenses and time needed for completing the process.
In order to plan everything properly, I made a detailed document, kept in the cloud, with the names of the protagonists, the gist of their stories, the scope of Skype recording, the description of the green screen actions and technical challenges for each scene. We developed more than 20 possible postcards in both types of exposition: off-screen narration and the connection of the two worlds. I also explored some other formal possibilities, such as long shots with excessive camera movement, interiors containing digitally added objects, locations moved to different environments, or the deconstruction of space. I also made relevant storyboards, concept art, and animatics.
This was, in fact, the second time I encountered the tension against crossing a technological barrier, similarly to the Skype recording problem. The director did not want to enter a certain technological level. He explained that it was beyond his working method and that he was afraid to lose control of the process. Instead, we tried the less advanced solution based on a system of hidden headphones and monitor projection, with the director prompting the protagonists during filming. Those solutions were tested, but the results were not satisfying for Ziemilski. He had a problem introducing natural behavior of the protagonists, because they acted in an unnatural way. The green screen made this way did not live up to the director’s expectations. The technology must serve the production team but the director found that the solution was confusing and it did not bring the expected results. This happened despite the fact that all the modifications of the method were done according to the director’s suggestions. The method required further development.
I was always fascinated with artistic activities based on image projections. In the late nineties, it was hard to find standardized equipment setup for VJing. Many gave up using it, but others wrote software to make it easier [5] In 2000 I created my first visual system using action script. It was prepared for the performance and the multimedia show I had in the planetarium. I kept improving this system during the next performances, making visuals for music improvisation. In the beginning, it was a live drawing system, but it evolved into the method based on live compositing and editing based method. At that time I had to rely entirely on my own creativity.
I was skeptical against using this solution for the documentary. Firstly, this method seemed cliché for our film. Projections are quite common in low budget realizations, especially in video clips. This topic has already been extensively explored. Projections in post-production are easy to replicate, especially when the camera is static. All you need is a layer with a proper merge algorithm and distortion. I believe that a good movie needs a new approach to the topic which was difficult to achieve with such a commonly used technique. Secondly, I knew that was impossible to receive high-quality shooting because the projection will be always degraded through the resolution and the brightness of the projector. In addition, the casted picture is distorted and irregularly blurred. In November 2016, I checked with the students the possibilities of evoking emotions through projection in the performance test during The Social Fantasy workshop made with cooperation with Black Box theatre in Oslo. This experience only strengthened my conviction that this solution can be just our plan B for emergency. Falling in love with one’s own tools is dangerous, and I worried that this method is already overused.
I revisited Iceland in March 2017. This time I was equipped with appropriate tools for a set survey. In this kind of production we need to provide a digital facility with detailed information about the live-action set. [6] This trip to gather data was essential for later creation. Right after the arrival, I found out that the decision was already made, and that we would make projection shooting instead of green screen shooting. This situation was difficult for me. I was prepared for a different kind of production, and the specialized equipment I brought was now useless. Gradually, I became used to this shift in plans. The director wanted to use my panoramas for the projections, so I started making them. I photographed more than fifty different locations fitting the stories told by protagonists. In addition, I made some 360-degree videos with the protagonists. They were designed in such a way as to multiply the moving subject and make him/her visible from several angles. The director wanted to use it interactively on set.
Some of the script for green screen scenes that we had already established were easy to adapt to the new solution. Projections are the analog special effect of the on-set type. We started to do further tests to check to what degree the casted pictures are visible indoors, when projected colored objects. We also checked the visibility of projections on the human body, the number of details one may notice, and the degree of distortion. We planned some additional effect shots, such as a nail painting scene. We got a night shot with some footage of waterfall at night. The production continued with classical shooting. Eventually, after more testing, I got accustomed to the new method of realization. This method was sufficient to show the relations between the subjects, it reflected our intentions. Projections allowed to achieve this result in an easy way. They were not only an upgrade from Skype, but something more, something which allowed to show a different kind of contact between people. In the communication of this type more happens inside the subject’s head than during the regular conversation involving two sides. It was a natural trigger for emotions. Maybe not everything went on exactly as I planned, but the result was equally good and provided an effective solution for the production problem. Finding a compromise and getting accustomed to it is one of integral parts of a creative technologist’s job.
The director wanted to control the position of projections during filming. I showed him an easy way to manipulate the picture through the Android-based panorama viewer. It used the gyro system of the smartphone for rotation and it was easy to set up. I tried to find a more sophisticated method of control, based on a joystick or a 3D mouse, but I had no time to develop it sufficiently. We had access only to an ancient, low-quality projector there but I was convinced that if these tests produced acceptable results, we could expect an even better performance with professional equipment. As Spinrad points out, projectors are basically just very simple, dumb monitors. They will take whatever you feed them and make it look big, bright and great. [7] My concern was the quality. The sharpness of the filmed projection was lower than the filmed scene. From half a meter distance, the projector’s matrix was clearly visible. Any picture distortion magnified this artifact. There was no way to improve it in the post production. Theoretically, there was a way to avoid distortion, but it required specialized software.
Unfortunately after this second visit to Iceland my time allotted for this project ran out, and I could not follow the rest of the production. I had to focus on my second project, a virtual reality presentation for the Philharmonic.
In August 2017 the production was continued without me. During panorama shooting in Poland, the director contacted me through Skype and asked me to prepare some footage for projections. I tried to increase the size of static panoramas, but the resolution of 27294 x 13095 pixels still delivered low quality in the camera and was too demanding for the equipment used on set. I saw it coming before, but I had no possibility to find a solution for it anymore. I also prepared a 360-degree video containing multiplication, but they had a problem with a large size of the file and with the codec in the context of the video player capabilities. Another problem was downloading 40 GB files through slow internet connection available on the set. Their rig consisted of the most advanced iPhone model attached to a big 50 kg projector and a camera on a heavy tripod. It was uncomfortable to operate it, and the gyro mechanism in the smartphone was inaccurate. I had never a chance to talk to the VFX supervisor who continued my work. It was a mistake, but I could do nothing about it.
The “postcard” development was the beginning of my experience with a spatial montage. Manovich claims that this type of montage within a shot includes the superimposition of separate realities which form contingent parts of a single image. [8] Green screen sequences and subsequent projections constituted this kind of superimposition of images. Such a technique resembled rear-screen projection shots or nineteenth-century avant-garde filmmaking experiments. The image created through keying represented a hybrid reality composed of two different spaces. [9] It was possible to achieve by means of contemporary VFX technology. In a similar way Zbigniew Rybczynski, a famous Polish filmmaker, uses in his films the combination of pictures complementing one another. Philosophically, the difference between the method containing projections and the one based on green screen resembled the difference in the kind of contact experienced by the protagonists on set. Instant SFX, which was projection, allowed to receive the result with two steps, while VFX required additional post-production, and the final result was visible only after completing this stage. The projections evoked more accumulated emotions, more directly and more immediately. In fact, none of the characters presented saw the finished movie during the four months after its premiere.
Looking for the form is closely related to the storytelling process. The story, however, should take precedence. A creative technologist supports the process of searching for the real topic of the film or new media realization. Everything that happens in the story relates to the central question which is asked in this story. [10] This search can be called looking for “the need” or “the protagonists’ needs”. The documentary should try to change something or to solve some problem. It is not only a good story. All my activities as the creative technologist focused on this problem, on finding the answer to this most crucial question: “why are we doing it?” When we clarified the “the need”, the form was more apparent. In our realization, the form had to support “the need”.
The main problem in that project was that I was not involved in the film from the very beginning to the end. It was impossible, considering that the realization took more than five years. Most of it was pre-production and development, both in Poland and Iceland. During my research, we developed the ideas that would be sufficient for several films. We tried many different techniques and methods, and eventually, we found the solution. The final shape was the director’s decision, but the film could not be the same without several artistic choices I made. The inductive method allowed us to join varied and broad plots into a harmonized story. To conclude, the film shows the true essence of the characters’ lives because we worked on both the visual side and the story. In my opinion, these two factors belong inseparably together. All of these elements must provide artistic unity.
