Check out in-depth details of advanced Visual Effects techniques by AliceVision.

We interviewed Fabien Castan (R&D Engineer, Mikros Image) regarding AliceVision and all its latest technologies. Detailed information are as follows. 

Our readers would like to know what is AliceVision all about.
AliceVision is a Photogrammetric Computer Vision framework providing 3D reconstruction algorithms. The project is a result of collaborations between academia and industry to provide cutting-edge algorithms with the robustness and the quality required for production usage.

Share with us descriptive information regarding Meshroom, the open source photogrammetry software.
Meshroom is a 3D reconstruction software based on AliceVision. It enables graphic artists to create textured 3D models from photographs.

We provide binaries for Windows and Linux. There is no official support for MacOS yet and there is a known issue on Win7. The software requires NVIDIA GPU and it is recommended to have a minimum of 16 GB of RAM to use it with the default values.

Check out official video of Meshroom: open source 3D reconstruction software.

Please explain in detail what is photogrammetry? Also let us know about Photogrammetry Pipeline.

Photogrammetry is the science of making measurements from photographs. It infers the geometry of a scene from a set of unordered photographs or videos. Photography is the projection of a 3D scene onto a 2D plane, losing depth information. The goal of photogrammetry is to reverse this process.

How 3D reconstruction is becoming very essential in production pipeline of major Animation and VFX studios?
Photogrammetric computer vision provides algorithms to mix real and virtual worlds in a single image. It is the key for Augmented Reality (AR) but it is also the root of all visual effects combining live action and computer graphics.

3D reconstruction is often used in large scale to create a survey of the set and provides support for the camera tracking of all shots in the same reference system. It also enables to create precise reconstructions for props modeling or set extensions.

Following image shows survey of the shooting set on ‘See You Up There’, Directed by Albert Dupontel, Production of ADCB Films, Manchester Films.

Image shown below shows survey of the shooting set on ‘Santa & Cie’, Directed by Alain Chabat, Gaumont Distribution.

Kindly explain complete process of Photomodeling.
First, it starts by a shooting session, which is by far the most critical step. You should be careful to correctly cover the object or the scene. The angle between 2 camera shots and the surface should be lower than 30° to match them correctly. As we use pixel intensities as the source of measurement, you should ensure to limit source of blur that drastically reduce the precision of the geometric information. So avoid motion blur (tripod is recommended) and limit the depth of field by using small lens aperture.

Then, put your images into Meshroom and launch the computation. It will generate a textured mesh automatically. In all standard cases, the default parameters will be optimal. The generated 3D mesh is heavy and contains artifacts or missing edges, so graphic artists do retopology to clean the results in zbrush, blender or maya. If you don’t need this level of quality, you can just decimate the mesh directly in Meshroom.

Finally, you can put the new mesh back into Meshroom to create the final textures from the input images. Meshroom will use the input mesh UVs texture coordinates (or generate them if no there is no UV).

Above output is the 3D reconstruction result of a wood statue from 115 photos taken with an iPhone. Left image is textured 3D model/wireframe and right image is geometry/textured 3D model.

Check out more results of our official channel on Sketchfab.

Please let us know about your various software and libraries.
Meshroom provides a user interface to enable graphic artists to create 3D reconstructions. The nodal interface allows them to iterate, augment a previous reconstruction with new images, re-texture a post-processed version of the mesh, and enables multi-machines parallelization for render farm computation. This nodal framework has also the ambition to provide a solution for computer vision researchers to analyze the impact of each step. Access to all internal parameters interactively and be able to replace one node by another approach and compare them in the same pipeline.

Meshroom is implemented in Python to allow us to evolve quickly and also to enable technical persons to easily discover the project and contribute to refine the user experience.

AliceVision is a C++ framework with some part in CUDA. This library provides implementation for multiple view geometry problems that can be useful in other specific contexts.

Why did you choose open source methodology?
AliceVision is the result of a collaboration between multiple partners:

• Czech Technical University (CTU) in Prague, Czech Republic
• IMAGINE from the Universite Paris Est, LIGM Gaspard-Monge, France
• Institut National Polytechnique de Toulouse (INPT), France
• Mikros Image, Post-Production Company in Paris, France
• Simula Research Laboratory AS in Oslo, Norway

AliceVision is released under the MPL 2.0 (Mozilla Public License 2.0) license, which allows usage in commercial software but impose redistribution of the source code modifications. These simple rules maximize the collaborations, contributions and impact of the project on both academics and industrials.

The scientific community is increasingly accepting the value of reproducible research with open source software, shared datasets, and generally, allowing others in the community to easily verify presented results and build on top of them.

In the industrial community, the photogrammetric computer vision technologies have impacts in many other fields than Media & Entertainment, like topographic mapping, architecture, engineering, manufacturing, quality control, geology, archaeology, scientific  investigation, etc. Making it open source is a way to join forces around common goals.

At Mikros Image, our business model is servicing and not software sales. Our added value is the relationship we build with our customers, our creative teams, and also with the software developers community as well as the research community.

What are your production credentials?

Mikros Image is a Visual Effects and Animation company founded in 1985 and is part of the Technicolor group since 2015. The studio is known for its visual effects on European films and commercial as well as more recently on animation films. Mikros Image is positioned across 3 business lines : advertising under the branding Mikros  MPC Advertising, film post-production and Visual Effects under the the branding Mikros Technicolor and animation feature films under the branding Mikros Animation.

Visit our website : http://www.mikrosimage.com to see our latest productions : “The Sisters Brothers” directed by Jacques Audiard and produced by Why not Productions, “Stronger” directed by David Gordon Green. For Commercials: Dior, Lacoste, Jean-Paul Gaultier ads amongst many. And in Animation “Captain Underpants: The First Epic Movie” directed by David Soren for Dreamworks Animation, “Asterix: The Mansions of the Gods” directed by Alexandre Astier and Louis Clichy for M6 films, and “Sherlock Gnomes” directed by John Stevenson for Paramount Animation.

We have been awarded a lot of prestigious prizes and awards (1 Oscar for Short Animated feature film, many Golden Lions for commercials, dozen of french Cesars for feature films, 4 VES awards, etc…) 

A large number of productions have been using our photogrammetry tools including : “Chocolat” a feature film directed by Roschdy Zem (production  Mandarin Films), “See You Up There”, feature film directed by  Albert Dupontel (production  ADCB Films, Manchester Films), “The Returned”, a TV episodic from showrunner Fabrice Gobert (production  Haut et Court, and Canal+), “Duracell” commercial directed by Günther Gheeraert (agency Grey London), “Santa & Cie” feature film directed by Alain Chabat (Gaumont Distribution).

What can we expect in upcoming versions?

The most important improvements will be on the display of intermediate data in Meshroom, performance improvements, support for rig of synchronized cameras and support for camera tracking within a 3D reconstruction. We currently have a lack of documentation and we will try to improve it as we go along.

The Virtual Assist thanks Fabien for this exclusive technical interview. Keep looking this space for more updates on advanced technologies.