Planning and shooting guidelines for immersive media video production

Shooting video for immersive media productions requires techniques and approaches that are not commonly used in traditional film work. Understanding these is crucial for creating effective, convincing VR that delivers a strong sense of presence.

Immersive media production differs from the production of traditional 2D media because in VR, audiences are surrounded by content. Instilling feelings of presence and immersion are important in VR storytelling.

Just like in traditional media, viewers will take on a certain perspective in a VR story. Here is a list of several possible perspectives in order of perceived involvement in the story from least involved to most involved:

• The true observer: the camera provides the viewer presence in the scene, but the viewer is invisible to, and ignored by, the main characters. This is how most traditional films are presented.
• The ghost: some characters in the narrative might acknowledge the viewer with a look, or overtly stay out of the viewer’s way, but they never interact with the viewer directly.
• The sidekick: characters in the story talk to the viewer as if they were a real person and present in the space. In these narratives the viewer is an active participant, which can be an effective way to deliver the feeling of playing a role in the story.
• POV: the viewer effectively assumes the role of another character. Sometimes creators will have a body on screen or a voiceover from the character whose point of view the audience is experiencing.

A classic example of poor perspective is an immersive story that takes place around a dinner table. Early on in immersive video production, creators often placed 360 cameras in the middle of a table, thinking that it would be immersive for audiences to spin around and look at everybody in the scene. The reality was that it was upsetting and confusing. Many viewers disliked having to turn around so much, and there is anxiety in being in the middle of a ring of people. Additionally, many folks felt like they were severed heads on the dining table! None of this would have been the creator's intent. A better approach would have been to place the camera at a seat at the table–one with a clear view of all the key characters. This would create a feeling of being part of the dinner party rather than the dinner itself. This kind of placement is fully compatible with all four of the different kinds of viewer perspectives listed above.

No matter which viewer perspective is chosen, creators should think carefully about their audience. It can be much easier to make creative choices that direct audience attention, keep them engaged, and focus them on story beats if audiences understand the perspective of the narrative and aren't worried about missing something that might be behind them (i.e., FOMO mitigation). In this way, thinking of immersive media as a kind of role playing for viewers can be helpful in designing successful narrative experiences.

Plotting out the action of your characters not only has technical advantages, but can have story advantages as well.

On the technical side, be aware of how VR cameras work. For example, when filming in 360 you might not want to have the focus of your scene in a stitch line. For monoscopic 360, this is the area directly between two lenses–the overlapping area may cause there to be artifacts in the final stitch. However, for stereoscopic 360, that overlapping area is actually the sweet spot for the best looking 3D, so cameras need to be rotated specifically to target the best result for the target distribution format.

When filming in 3D-180, there is a sweet spot of really good stereo in the center of the frame, which is the ideal place for action, since it delivers the cleanest shots and is the most comfortable to watch.

There are also proximity concerns that vary depending on the camera system used. In stereoscopic shooting, creators need to think more carefully about the subject distance from the camera: too close and the stereo effect is physically uncomfortable due to a visual phenomenon called ‘vergence-accommodation conflict’; the eyes are crossed to converge on a close subject (‘vergence’) while being focused on the virtual image (‘accommodation’). If subjects are too far, viewers might feel disconnected from the story. Since we can’t zoom into an immersive scene the way we can in traditional media, anything far away tends to lack detail and relative impact. In addition, VR headsets might not be high enough in resolution to support small subjects in the field of view.

During blocking (the staging of actors and subjects during planning), understanding technical aspects of production should also inform creative choices. Typically, close subjects moving through stitch lines can cause distracting visual artefacts as a result of stitching between lenses. While these are possible to mitigate to an extent in post production, this requires both time and budget. Knowing your camera's capabilities and the details of post-production workflow can help prevent complications during post.

Planning the movement of characters, especially in 360 storytelling, is required for guiding viewers. Audiences generally follow the movement of characters in the scene, so if you want your audience to turn left, consider having a character walk from right to left to guide the viewer towards that direction. When cuts are made, viewers should not have to spend time placing themselves in the new scene; there should be a natural subject for viewers to focus on in the expected view orientation. If there is too much movement in the same direction from shot to shot, you'll cause viewers to turn around completely. This may not be comfortable for those watching while seated in non-rotating chairs. Taking these considerations into account will help prevent your content from causing too much discomfort, and will aid in keeping your audience immersed.

Unlike VR games and other virtual experiences that are in six degrees of freedom (6DoF) , immersive video only allows rotation during viewing–it is three degrees of freedom (3DoF). Immersive videos are shot and presented from a fixed point in space, and because of this, camera height is very important and should be given careful consideration.

A camera at the eye height of an adult is likely to feel a little high in VR, in part because the majority of the visual interest is then at or below the horizon (this is something that most VR directors have learned through experience). A lens height of between 62in and 66in (1.57m to 1.68m) is a good starting point. Many directors center lenses at around the chin height of the main character, but it is important to actually test how this feels in a headset.

Maintaining a consistent camera height throughout a production is good practice, although you shouldn’t feel completely forced to do so. It can be strange for viewers to be at slightly different “standing” heights throughout a video, but this may not apply if you are intentionally moving from standing to seated, or if the production involves placing cameras in unique locations where there would be no expectation of normal or consistent human perspectives. As always, testing in headset is important.

Movement of the camera in immersive media should be considered and done with care. Since the viewer isn't physically moving, any acceleration of the view perspective will cause some amount of discomfort in viewers, and in serious cases motion sickness and extreme nausea. Note that all rotations are accelerations and rotations are considered to be extreme camera movements in VR production. If it is important creatively to move the camera during filming, here are some considerations.

1: Minimize acceleration. Acceleration is the thing that causes motion sickness, not movement itself. A camera moving linearly without accelerating (and without rotating) can be comfortable for viewers, but when possible, cuts should be made into movements that are already at speed (as opposed to say, cutting to a stationary camera that they begins to move, which is an acceleration).

2: Forward movement is the most comfortable. There is debate about this point, but in general, forward movement is considered to be the safest in immersive videos. Backwards or lateral movements can be a little disorienting for some, but all movements should be tested in headset.

3: Stable movements are required. All camera movements should be stabilized. Without stabilization, the results are almost guaranteed to be distracting, uncomfortable to watch, or nauseating. In 360 video production, pitch and roll stabilization can be done effectively in post, although the vertical ‘bounce’ caused by unstabilized walking can’t be corrected. In 3D-180 production, stabilization during shooting with the use of tripods, rails, gimbals or other stabilizers is required, as post-production stabilization is challenging and reduces the field of view.

4: Maintain a steady horizon to help prevent motion sickness. There is no need for you to tilt your camera to look at something; the camera captures the complete scene from top to bottom, and your audience will be able to look up or down when viewing. If you do want to tilt the camera up or down, it should be done with strong creative intent, and should be tested in headset. When capturing stereoscopic footage, cameras must be kept absolutely level because people will automatically adjust their head position to make the horizon level. In headset, this natural human correction creates a stereoscopic vertical offset that is very uncomfortable (there is an exception in cockpits–see below).

5: Avoid panning. Panning, or rotating the camera, is an aggressive acceleration and is almost guaranteed to cause nausea. Avoid panning at all costs. If a pan somehow ends up in an edit, one way to mitigate is to “blink” the scene during the pan (quickly dip to black during the pan).

6: If movement is unavoidable, the ‘cockpit effect’ can help. The cockpit effect relies on a static environment surrounding the camera, such as the cockpit of an airplane, and all of the motion in the scene occurs outside this static environment. This helps viewers to have a frame of reference during movement, and is also part of how ‘4D’ rides at amusement parks reduce nausea when cameras move. For example, when inside a moving car, the outside scenery may be moving by quickly, but the footage can still be comfortable to watch because the car’s interior is static in the frame.

7. Vignette or blink. During editing, if a clip’s motion is a problem, try vignetting (reducing the field of view with a feathered black mask) or dipping to black like a ‘blink’ while cutting out the worst offending movement. A vignette is commonly used in first-person games in VR during controller-based movement.

Capturing high-quality audio on set makes mixing during post-production significantly easier.

Consider using an ambisonic microphone on the camera to capture a spatial recording of the environment. It is also recommended to roll at least 1 minute of environmental sounds without any dialog as a sonic background plate. This is known as ‘room tone.’

Ambisonic microphones should be placed as close to the camera center as possible; they will essentially be your "ears", and it feels strange to the viewer to have the location of their ears disconnected from their eyes. It is helpful to know the ‘north’ or front orientation of your spatial audio microphone and ensure this matches the center of your camera image. Otherwise, an audio engineer will need to try to figure this out during mixing, which is a difficult task. Misaligned audio can be disconcerting to the viewer.

For the best-quality dialog recordings, avoid recording the talent with only an ambisonic microphone. Ambisonic microphones will be located away from the speaker and can pick up a lot of the reflections of an interior room location or exterior noise such as traffic, airplanes and other environmental sources. The greater distance between characters and an ambisonic microphone can also make it tough to isolate their dialog from the environment sounds in post production.

Using shotgun microphones on boom poles in 360 immersive production can be complicated or impossible because it is hard to get the shotgun microphone close enough to the talent without a lot of painting and rotoscope work to remove the microphone in post production.

Note that high end ambisonic microphones can emulate any microphone polar pattern (including ones that resemble shotgun mic patterns), but this generally requires a knowledgeable sound engineer to be involved.

If working in 3D-180 and the subject is fairly close to the camera, it’s possible to use a shotgun microphone, but even then getting close enough for the cleanest audio is hard. The further away the microphone, the wider the pickup pattern is and the more sounds it will pick up from the environment. If there is too much ambient sound in the recording it can create problems during the audio mix because background sounds may be spatialized incorrectly in relation to the main dialog.

Generally, it is a good idea to have very-close mono recordings from either lavalier microphones or hidden plant microphones. These mono sources can then be spatialized in post production.

Since audio is almost always externally recorded, it’s important during production to slate each shot with a verbal cue and clap (preferably with a slate, also known as a clapperboard). For example, someone would say “Scene 2, take 2,” and then give a single clap. Be sure the “north” or center camera can clearly see where the slate or hands hit. If the camera records audio, leave it running as a scratch track, which also will make syncing easier. Timecode slates aren’t generally needed, as most VR cameras do not record timecodes

It takes a lot of equipment to create high-quality and cinematic immersive media. Visible equipment in a final piece can reduce the sense of presence and immersion that makes VR compelling.

To remove unwanted elements such as lighting, crew, and (most-commonly) the tripod or monopod the camera is mounted to, VFX clean plates should be captured for every camera position. A VFX clean plate contains the visual information needed to composite out unwanted elements.

For example, let us say we have a scene in which the director is standing behind the camera in order to interview and interact with a documentary subject. After the interview is over, without moving the camera, a VFX clean plate can be captured by recording an additional take with the director no longer standing in the same spot. In post production, the two scenes can be layered on top of one another and masked to show or hide various parts of the scene. Effort should be made to capture VFX plates in the same lighting conditions as the original takes.

When dealing with camera supports such as tripods and monopods, it is often possible to digitally paint out the bottom, or nadir, but having a reference photo is helpful. A photo shot from the same height and position of the VR camera, but facing down to cover where the tripod/monopod was, usually is sufficient for nadir patching in post production. Even a photo taken with a cell phone is better than nothing at all as long as the entire area once occupied by the tripod is covered.

When shooting a VFX plate for this kind of use be aware of changing lighting conditions and shifting shadows.

Lighting in immersive media can be tricky for many reasons. In the 100+ years of 2D rectilinear video production, many standard lighting techniques have been developed to create “cinematic” looks. Unfortunately these often require a lot of visible gear that are not easily compatible with immersive capture, and immersive media production requires different techniques,. At the same time, most VR cameras use smaller, noisier camera sensors that give less flexibility in post production, so overall, cinematic lighting is a challenge for immersive video production. The following techniques and guidelines can help you create a more cinematic look:

• Capture in locations with naturally cinematic lighting. Look for interior spaces with large windows and natural light. When capturing exteriors, try to shoot either during the morning or late afternoon; avoid the middle of the day when the sun is at its highest. A high sun usually results in harsh lighting and unflattering shadows.
  
• If using the sun as the primary light source, block your characters so the sun is lighting them from an angle or slightly behind them for a more cinematic look. If the sun is at the camera's back it tends to flatten the subject, and it also casts a camera shadow towards the subject that will be difficult to remove during post production.
  
• Use practical lights when possible. Practical lights are things like normal lighting found in a home designed to be seen by the camera, and not studio/filmmaking-specific lighting that is undesirable to leave in a finished video. If filming indoors, replace the lights in the location with the same or similar bulbs: same color temperature, same output, and if possible, dimmable. Bring in extra lamps as production design to help shape the light of the room. Having extra-bright bulbs on hand helps to provide greater practical illumination, especially for the smaller sensors commonly found in VR cameras.
  
• A small light on the monopod or camera can function as an eye light. It's important to give a little fill to faces, particularly for eyes. Don’t use a strong light for this; it's generally not a good idea to have the key light in a scene close to the camera as it will flatten out subjects.
  
• You may need to augment practical lighting with traditional film lighting and grip gear. These will need to be removed in post, so remember to capture a VFX clean plate. A clean plate is made by removing or moving the lights to a different part of the room so that the camera can capture an additional shot of the scene without the lights. This allows for compositing the two shots together in post production in order to remove the unwanted film gear from the set. Lighting for 3D-180 is often much easier because the whole back half of the scene is not captured, which means some traditional production lighting practices can be used.
  
• Using large and soft light sources that directly supplement a natural light source can make the lighting design process easier. For example, the light coming through a window is not bright enough to illuminate the subject, consider placing a higher powered light near the window that casts light in the same general direction. When the light is removed in post production (using the VXF clean plate technique), audiences will assume the light source is from the window.
  
• It's important to place the lights in positions where characters won't cross, unless you are planning on rotoscoping in post production. This means hiding cables and stands, and often, lighting the scene from above. For example, you could mount a long metal tube known as a ‘speed rail’ between two walls using a wall spreader up by the ceiling of a location to provide overhead mounting points for lights.

Obviously, it breaks any possible sense of immersion when audiences see crew members in the scene, so it is important to think about where to stage your crew while filming. Usually, only essential personnel should be near the camera when filming. They should be grouped together as tightly as possible. Consider keeping them behind the director in a single line.

Find a safe spot to stand. Even on location there is usually a safe spot for at least the director to be so they can evaluate whether or not a take was good. Low, near the camera, is usually a safe spot, as are in front of flat and shadowed surfaces such as walls where characters don't cross and the environment doesn't change. This helps keep the process of using VFX clean plates in post production relatively straight forward.

Ideally, the director should never have to hide. However, if there are no workable options to shoot a VFX plate, then everyone must be out of sight. If this is the case, a camera monitoring system is helpful for remote viewing, and a way to communicate with the talent need to be established.

Always shoot a clean plate if possible. Don't trust that your entire crew has hidden themselves perfectly! After the scene is complete, record a take that is completely empty or has all of your crew standing on the opposite side of the camera so you know you have clean plate coverage.

VR cameras often have multiple fisheye lenses. All lenses need to be completely clean and free of dust, smudges, or other imperfections. Smudged lenses can cause nasty glares from lights and make content uncomfortable to watch, especially in stereoscopic content. Lenses should be checked with a flashlight to ensure that they are clean.

Be aware of lens flares when shooting. Although lens flares are used in traditional media for effect, in VR productions they are usually a problem. It is a good idea to view each lens individually before filming to ensure that the image it captures will be clean and free of flare. When stitches, lens flare can cause visual discomfort or introduce odd stitching artifacts that are very difficult or impossible to remove in post production.

A camera operator should always have the following items:

• A can of compressed air
• Lint-free wipes such as lens cleaning tissue paper
• A blower (Such as a Giottos Rocket Air Blaster)
• Lens cleaner fluid specifically formulated for lenses, which is important to avoid damaging delicate lens coatings.