After joining Google and Huawei at the UW Reality Lab at the University of Washington in January, it became apparent that Facebook is already recovering its donation.
According to results released earlier this month via Cornell University's arXiv service in Cornelia, an article titled "Photo Wake-Up: 3D character animation from a single photo" has a team of Facebook and University of Washington researchers developed a method that can be used to create animated 3D models in augmented reality.
The dynamics work by fitting a morphable body model to a Picture and guess a body map of this model. With the associated model, the system creates a 3D mesh, applies textures to the mesh that match the body map, and integrates a skeletal platform to control the movement of the figure. From this reconstruction, the system can extract a 3D figure from a photograph. The research team claims the system works with photos, cartoon characters and even abstract paintings.
The demo video for the system shows a HoloLens user watching a figure of a Picasso painting fleeing the screen, an extreme sportsman bursting out of it, and Paul McCartney stalking out of the Beatles' album art Albums Help!
The team released its findings on the arXiv service Cornell University in an article entitled "Photo Wake-Up: 3D character animation from a single photo". Facebook researcher Ira Kemelmacher-Shlizerman, also assistant professor at the Paul G. Allen School of Computer Science at the University of Washington and founder and co-director of UW Reality Lab, co-author of research with Brian Curless. Professor at Allen School and director of the UW Reality Lab, and Chung-Yi, a graduate student at the university who worked as a research associate on Facebook in the summer of 2018.
Kemelmacher-Shlizerman joined Facebook in taking over her startup, Dreambit, and her research on moving portraits is now on Google. Curless's 3D surface reconstruction research contributed to the environmental mapping algorithms in Microsoft HoloLens and Google's Project Tango. Previously, the faculty members of the UW Reality Lab had already been involved in a similar project in which football videos were reconstructed to render augmented reality replays on tabletops.
As mentioned in the limitations of the section, the results are somewhat rough in practice (which is indeed the case) (in favor of abstract art), but the computer vision approach is in line with Facebook's research in AR full body masks and the image recognition and other capabilities of the Spark AR platform.
However, the research findings are as they are. The team has demonstrated the AR experience only through HoloLens, although the description of the method does not specify that the depth sensors of the headset are required.
However, the technology may not immediately find its way onto either the Spark AR platform or the Google mobile AR efforts ultimately benefit from the research made possible by their donations to the UW Reality Lab.