For those football-lovers in our audience and especially the tech-oriented ones, get ready for an amazing TV experience of your favorite football team (I hope it’s the Jets) created by Intel’s amazing new True View system.
Here are the basics of the technology:
There are more than 30 5K cameras (This link is an example of a Pro 5K camera but may not be the ones used in the Intel system) which surround the stadium or other venue, covering every inch of the playing surface with amazing depth. The cameras capture volumetric data (height, width, and depth) using voxels, a 3D pixel. This technique uniquely renders spectacular 3D replays and creates a multi-perspective view of key moments.
Processing power —Intel True View image capture generates a staggering amount of data and requires incredible computing power. On-site Intel server farms process the volumetric data used to create 3D replays.
Proprietary Data Algorithm—Powerful Intel servers and a proprietary data algorithm process data at remarkable speed. The result is a seamlessly rendered Intel True View replay.
Live captured content for live captured 3D data from multi-camera set-ups or RGB-Depth cameras needs to conform with both, requirements for 3DTV or 3D-telecommunication systems, in particular narrow-baseline high-quality 3D capture and more wide-baseline oriented Free-Viewpoint Video (FVV).
IBM has developed this client-server architecture which can stream 3D multi-sensor data of a person captured from different angles which will provide a stereoscopic free-viewpoint experience for virtual reality applications. The capabilities of this system are featured on view-dependent Virtual Reality (VR) pipelines such as a VR-HMD or on an auto-stereoscopic display output device. See Figure 1.
An overview of the View-Dependent Streaming Pipeline (Image courtesy of Reference 1)
The process begins with a camera array that captures volumetric (height, width, and depth) data of all the action. The data captured is analyzed, reconstructed, compressed, and encoded before it is stored. Voxels (pixels with volume) allow the technology to render replays in spectacular, multi-perspective 3D.
Every installation includes 30—50 high-end 5K JAI cameras, which will give a full field cover in higher-than-HD resolution. They are distributed around the stadium's perimeter and target specific areas to ensure full-stadium coverage. Cameras are connected to the server room through fiber optic cables.
Each Intel True View system requires more than 50 servers. They include Gigabyte X-99 motherboards with an Intel Core i5 processor. Servers are capable of processing up to 1 terabyte of data per 15—30-second clip.
As the 3D processing server (yellow rectangles in Figure 1) merges the input data and process a created and compressed 3D + color representation of the scene, the rendering client (blue rectangles in Figure 1) recreates the transmitted scene portion. In essence, the server reduces the amount of transferred data by only sending the data necessary for an optimized view-point.
In the image acquisition part of this process, the architecture uses two RealSense R200 sensors that have a resolution of 640×480. These sensors output a color plus a depth image. (These cameras are now discontinued since the initial developent may have been upgraded by a newer technology, possible the RealSense D400 family of ‘depth cameras’).
See Reference 1 for more in-depth details of the initial IBM design for this system.
Watch the video in the link below:
1 A CLIENT-SERVER ARCHITECTURE FOR REAL-TIME VIEW-DEPENDENT STREAMING OF FREE-VIEWPOINT VIDEO Jonas Scheer, Carlos Fernandez de Tejada Quemada, Saarland University, Intel Visual Computing Institute, Germany, and Oliver Grau, Intel, Germany IEEE 2017