The ceiling and walls of the new Canon Volumetric Video studio in Kawasaki, Japan. The walls are green and have small, round black objects protruding from them at seemingly random positions and intervals. Above are more than two dozen flat square lights hanging from a rig below the ceiling.

Watching from within at Canon’s volumetric video studio

We, the audience, are used to watching from the periphery. Whether it’s an onstage performance or our favourite team on the pitch, we are, of course, rapt – but usually from a distance. “Of course,” you say. “Where else would we be?” It’s a great question, and one that Canon Inc. now has the answer to:

Everywhere.

Less than an hour outside of Tokyo, Canon’s Kawasaki office is the home of a new ‘Volumetric Video’ studio. It is here that one is able to see the incredible potential of a technology that is capable of putting the audience everywhere. In the green-screened studio, real-time footage can be captured from any angle, then used to generate a reconstruction in 3D almost instantly. Together with Canon’s Free Viewpoint Video System (which in the near future will allow viewers to control how they see the scene), it could change the way we view live action forever. With it you could watch your favourite band from any point, travelling around the stage as naturally as if you were walking. Or have the perfect vantage point of a spectacular goal without any switching of cameras, as was the case when the Free Viewpoint system was put through its paces during an international rugby match in Tokyo during 2019.

The new studio opens up a world of volumetric content in some really exciting new areas, such as TV commercials, music videos and events. Not only can it take you anywhere inside the action, in almost real-time, but the fact it can also do so in 3D means that, in the near future, viewers can use VR or AR to immerse themselves in live action entertainment. Extraordinarily, the same technology could also render live scenes as holograms. It’s a glimpse of a future that turns traditional viewing inside out, and through it the audience gains free and open access to any part of the ‘stage’ while the ‘fourth wall’ between audience and performers remains intact. It feels fresh, dynamic, natural and holds endless potential for realising new ideas and reaching new audiences.

Two women in lilac jumpsuits stand in a green-screened studio. They each have an arm outstretched. Behind them is a man standing behind a DJ booth that is labelled ‘chelmico’. © Warner Music Japan Inc.
Conventional Camera: Japanese rap duo ‘chelmico’ were the first to livestream from the new studio, racking up just under 48,000 views on YouTube. © Warner Music Japan Inc.
The two women in lilac jumpsuits now stand in the same position in the same studio, but with red, grey and purple patterns on the wall behind them, and two banners with the word ‘maze’ on them. © Warner Music Japan Inc.
Free Viewpoint Video: Fans were able to watch Rachel and Mamiko live as they talked to their audience and performed tracks from their new album ‘Maze’. © Warner Music Japan Inc.

In a year when live events were sadly limited. the first project to be held at the new Canon studio was a Covid-secure livestream event for Japanese rap duo ‘chelmico’. Directed by Jun Tamukai, rappers Rachel and Mamiko were celebrating the release of their new album ‘Maze’ and used the studio to create a ‘virtual live show’ for fans. The broadcast alternated between conventional cameras and Free Viewpoint video, and viewers on YouTube saw the pair from multiple different angles – including above and below – that transitioned between 2D and 3D. They also had a peek behind the scenes, watching the duo manipulate footage of themselves with the viewpoint controls.

This incredibly fun spectacle was created with 89 4K cameras, all network connected and ready to simultaneously capture the same scene from multiple viewpoints. The images were then used to ‘construct’ high-detail 3D spatial data, using Canon’s proprietary image processing technology. The latency from image capture to Free Viewpoint output is tiny, and it took just a couple of seconds to process Rachel, Mamiko and their DJ into a 3D space. After which, a 3D camera then took over, which was freely moved around inside the 3D space to produce the 2D video content for broadcast.

A diagram showing an overview of the Free Viewpoint video process. L-R: input of stadium or studio activity. Images are captured by a video generation server and then either turned into 2D-rendered video for use in instant replays, for analysis or entertainment/music videos. 3D-meshed data can be output through AR, VR or hologram.

Of course, the chelmico broadcast was just three people, but the spacious studio ‘capture area’ can both accommodate multiple people and many subjects moving at speed, while still supporting a vast range of video production needs. For example, the simultaneous 60 frames per second (fps) capture of a fast ten-person dance performance can take place at the studio and, again, this creates a 3D data model of all performers as they move, from all angles. This in itself has exciting implications for theatrical and scripted content – after all, who doesn’t want to get their own perspective on their favourite shows?

Putting the viewer in the driving seat in this way may feel radical today, but this exciting tech has the potential to take many directions and give us new ways of experiencing filmed content. It’s going to be truly fascinating to see how creative brains push the envelope of this volumetric technology and, as they do with so many of Canon’s innovations, create truly transformational experiences.

Learn more about Canon’s Free Viewpoint Video System and the new Volumetric Video studio.

Written by Marie-Anne Leonard


Related Articles