Virtual Production：Past, Present, and Future(1)

The rising expressive power of Real-time rendering Jon Favreau's biggest concern about The Mandalorian is whether the game engine can reach the visual effect rendering level of Star Wars on the TV budget. But when he saw a piece of sand similar to episode 2, he abandoned the thought. Sandy environment is built with assets scanned with photos and rendered in real-time In the virtual studio, all the scene elements and lights can be switched and upgraded freely, and the feel of virtual reality is maintained. Part of the virtual environment of The Mandalorian directly uses assets created by Unreal Engine for some video games, saving a lot of asset construction time and opening up the possibility of asset sharing between the two industries. Real-time rendering has been able to better present the highly reflective materials often involved in science fiction dramas The virtual scenes on the set sometimes deceive the eyes of the people on the scene. Jon Favreau said, "There was a person who came to the studio and said,'I thought you wouldn't build the whole scene here.' I said, no, we didn't build here, in fact, there only have tables.' Because the LED wall rendering is based on a camera, there is parallax. But even if you look at it casually next to you, you will still think that you are watching a live-action scene. " Entering the virtual studio, the actor has entered the virtual world and the final scene The changes of Hollywood led by virtual production technology In addition to accelerating the cycle and increasing the turnover of the budget, virtual production has also brought revolutionary convenience to actors and other teams. Actors can see the situation of the environment in real-time to perform and interact. "This not only helps the cinematography but also helps the actors understand the surroundings, such as the horizon. It also provides interactive lighting", Jon Favreau described it as “a huge breakthrough”. Virtual production can also help fantasy characters interact with real people In addition to The Mandalorian, movie projects such as Lion King and Fantasy Forest have already begun to use game engines to produce movies, but they are still mainly in the visual preview stage. Besides, Steven Spielberg and Denis Villeneuve used Unity to help achieve their visual effects in the production of Number One Player and Blade Runner 2049 respectively. This method is gradually replacing the usual storyboard production method similar to hand-drawn comic books. Spielberg stood on the monitor to view the real-time synthesis of the shooting content In the virtual production process, VR technology is rapidly becoming a viable option for large studios and production companies. Director Jon Favreau used a lot of virtual reality technology in the recent remake of the live-action movie The Lion King, which fundamentally created the entire virtual reality world of the movie. Lion King uses a lot of real-time virtual preview technology The popularization of virtual production technology In terms of budget, virtual production and LED display equipment are still relatively expensive at this stage.　　 Complete real-time compositing and film production at a lower cost Jon Favreau indicated that virtual production technology is a major leap forward in the film industry, allowing creators to make creative decisions before the project starts, rather than reinventing the process or after completion. Nowadays, more people can see the appearance of the lens directly during cinematography. More people can contribute their own ideas and learn from each other because they can see the final idea in advance. "In the past, when you went to a scene and left the green screen, you no longer care about it. But now, we have so many talented people and have accumulated more than one hundred years of filmmaking experience. Why should we give up just because we change the shooting system? Let’s continue to inherit the skills of film artists and develop tools that adapt to the times.” Virtual production technology will gradually become the routine process of film production in the future

The Evolution of Virtual Production Virtual production applied in the industry in early times with less coverage. In the early 1990s, Hollywood used virtual production to make dynamic storyboards, which is the current Previz. 360 degree super static instant short technique used in The Matrix Titanic was released in the United States on December 19, 1997, followed by Matrix in 1999, Matrix 2 and Matrix 3 in 2003, and Transformers 1 in 2007 and its sequels. These films had made a good application and promotion of virtual production, especially the use of Motion Capture and Motion Control. Combination of motion capture and real-time rendering Virtual production is well known to the audience after the release of Avatar in 2009. On December 16, 2009, Avatar and its Pandora brought the audience a beautiful feeling, making the virtual production familiar to the public. The director can directly see the compositing scenes instead of the traditional green screen in the monitor, which makes it refreshing. Avatar takes virtual production to the mainstream When the whole movie is shot on a green or blue background, there are more than 2000 scenes need to have post-production, including editing, VFX, sound and others; and the staff needs to match the shots one by one according to the script description. Hence, demands have driven the development of technology. After Avatar was completed, virtual production was integrated into a commercial module. The Outlook of Virtual Production Although there are many technical problems that need to be improved for virtual production, the development trend has been established and will be prosperous with the rising trend of the global box office. The live version of The Lion King fully uses virtual production The driving effect of Hollywood VFX films has made more and more directors and filmmakers recognize and contact this field, and use this new technology in their creation. A material library for virtual production is needed in the near future. Cultural and historical monuments, natural sceneries, and others have already become frequent visitors to the movie screen. However, all cinematography must go to the site at a high cost. The schedule of the actors is also difficult to coordinate. Real-time compositing scenes in the monitor Virtual production can solve the problem and reduce production costs. For example, when shooting the Colosseum in ancient Rome by virtual production, the actors perform in the virtual shed, directly utilizing the scene of the beast in the material library for cinematography. Movies are the art of creation. Using photos as a background can not meet the need for cinematography technology. A three-dimensional model needs to be built with real textures. Light simulation, such as early morning, dusk, or noon effect, can also be performed as needed. Then put them into the model library, transfer to the virtual production system according to the needs of creation, and let the camera flexibly choose the required angle to shoot according to the needs of the script. The establishment of the material library is a huge digital project, which not only includes all kinds of materials but also enables clear retrieval and calling. This project requires continuous investment and construction. We can build a framework through special groups, formulate technical standards, and incorporate the conformity into the database. We can also commercialize it for a fee to make this material library enter a healthy development track and grow slowly. Only green screen props are needed to simulate the interaction with the virtual scenes In the future, more and more traditional studios will be transformed into virtual studios. Within the scope of human control, 24 hours of cinematography can be provided and will not delay due to any weather. Here is a common sense of virtual production. The scenes built by traditional studios are fixed. The replacement of reality scenes takes one day or a few days, but that of the virtual scene only takes a few minutes. In the performance area, the modules can be assembled outside the venue in advance in the studio. The production of Disney's Mandalorian brings virtual production into the LED era The actor's schedule is very tight. Any accident, such as illness, injury, and so on, will disrupt the normal cinematography plan. The virtual production will make the scenes in the script in advance, and quickly call different scenes according to changes, coordinating the process of the crew to make filming more humane and more efficient. Virtual production has a very good prospect. The continuous improvement of virtual production builds a proper platform to improve the quality of filming. Creativity can be presented in a better way, bringing audiences a better viewing experience.

The Birth of Virtual Production From the perspective of time and popularity of the application, the application of TV should be earlier than that of movies. In 1978, Eugene L. proposed the concept of "Electronic Studio Setting", pointing out that future programming can be completed in an empty studio with only personnel and cameras, and the sets and props are automatically produced by electronic systems. Virtual studio technology widely used by current mainstream TV media After 1992, virtual studio technology realized. As a new technology, the virtual studio became a hotspot in TV studio technology. At the IBC exhibition in 1994, the virtual studio technology debuted in various TV broadcasts. Virtual background can be changed with the lens in the virtual studio The Virtual Studio System (VSS) is a new TV program production system that has emerged in recent years with the rapid development of computer technology and chroma key. F1 program presents a demonstration effect through virtual scenes and props In the VSS, the position and scenes of the cinematography are transmitted to the virtual system in real-time. The green screen (or blue screen) is cleared by chroma key to replace the pre-made virtual three-dimensional space model. Then the host or actor of the picture emerges with the three-dimensional virtual scene into a new picture. Finally, the compositing video can be displayed on the TV in real-time. Using virtual studio technology for remote virtual dialogue Actually, the virtual scene does not exist in reality, but the technology that seamlessly integrates with the real shot characters makes the virtual studio a reality. The world's first reality show using virtual production