The 2018 FIFA World Cup tournament is happening this month and the Summer Olympics are coming in 2 years, which means it’s a perfect time to check out this paper that was recently presented by Konstantinos Rematas at the Conference on Computer Vision and Pattern Recognition – better known as CVPR 2018.
Soccer On Your Tabletop
(by Konstantinos Rematas, Ira Kemelmacher-Shlizerman, Brian Curless, Steve Seitz)
“We present a system that transforms a monocular video of a soccer game into a moving 3D reconstruction, in which the players and field can be rendered interactively with a 3D viewer or through an Augmented Reality device. At the heart of our paper is an approach to estimate the depth map of each player, using a CNN that is trained on 3D player data extracted from soccer video games. We compare with state of the art body pose and depth estimation techniques, and show results on both synthetic ground truth benchmarks, and real YouTube soccer footage.”
This paper, by a team from the UW Reality Lab and the Graphics and Imaging Lab (GRAIL) has received significant press coverage, for example, by Forbes (1), Forbes (2), The Daily Mail, Science, New Scientist, Digital Trends, GeekWire, Nvidia Developer, Tech Crunch, Inverse, King5 News, iProgrammer, ZDNet, and others.
This research project takes a simple, one-camera soccer video and turns it into a 3-dimensional game that you can view on a tabletop with an augmented reality viewer, like a Hololens. It accomplishes this with a deep neural network, and with depth and pose estimation of the players, all trained on synthetic data gathered from the FIFA 2017 video game.
There’s still work to be done before it will perfectly display a full game. The current version does not include the soccer ball, and it has trouble recognizing players when they are standing so close together that one blocks the view of the other. Once the researchers have made those improvements, it’s only a matter of processing speed to make it possible to watch a complete game in 3D on your table. And once this is complete for soccer, Rematas says that it can work for basketball, tennis, and other sports as well.
