Madonna rocked the stage with latest technologies of Augmented Reality and Unreal Engine.

 

Madonna needs no introduction. The giganticly successful pop icon is back with her latest album, titled ‘Madame X‘. But, it was a huge surprise for the audience when she performed her latest single track “Medellín”, with Latin singer and songwriter Maluma (Juan Luis Londoño Arias), on the stage of Billboard Music Award (BBMA) 2019, inside the MGM Grand Garden Arena, Las Vegas, Nevada. She was looking stunning, wearing the black eye patch with ‘X’ written on it. Madonna preformed with her four digital avatars aka virtual versions (secret agent, musician, cha cha instructor and a bride). It has been first of its kind. The core backbone of these visual extravaganza was:

• Augmented reality (AR) technology
• Volumetric captured videos / 3D videos
• Brainstorm software (InfinitySet, Brainstorm’s VR and AR application)
• Unreal Engine

Well, the final output / amalgamation of these cutting edge technologies is not visible to your naked eyes. The way you wear 3D stereoscopic glasses to ‘feel’ the effects of 3D stereoscopic movies, you need to see the AR output through a lens; either camera, TV screen or mobile. For the performance of Medellín, Madonna used on-set special effects with combination of Augmented Reality effects. The technical crew generated on-stage special effects of digital rain, cloud, scenery and many other pyro effects. These were seamlessly blended with Madonna’s four digital avatars, real Madonna, Maluma and dancing group. It was visual treat to see four Madonna characters (secret agent, musician, cha cha instructor and a bride) appearing and disappearing on the stage, with complete sync of real dancers.

So, in nutshell, 

• With naked eyes, the audience can see only real humans – (flesh and bone) Madonna, Maluma and dancing crew.
• There were video screens in the area. It was powered by real time camera tracking to combine 3D visual effects into the live dancing, with help of Augmented Reality (AR) technology, Brainstorm and Unreal engine. If you watch these screens, you will see real time integration of Madonna’s four digital avatars, with all computer generated visual effects, combined with real Madonna, Maluma and dancing crew.

[Tweet “Madonna’s augmented reality performance of Medellín has cost around $5 million.”]

How Sequin shoot the AR experience of Madonna’s Medellín?

Sequin, the AR company, took this challenge of incorporating the real world with digital world, real time visual effects in augmented reality. They made four digital doubles / virtual versions of pop diva Madonna – secret agent, musician, cha cha instructor and a bride. The biggest challenge of Lawrence Jones (co-founder of Seqin, executive creative director) was to create a truly amazing broadcast AR experience, with tight integration of 3D volumetric data, real time camera tracking and rendering the entire scene through Unreal engine. He has to match the choreography of real stage people with four 3D digital characters of Madonna.

Such projects, one of the best AR experiences, pushes the envelope for next level technology revolution for various Media and Entertainment segments. It opened up a new horizon for real time VFX integration in broadcast media. The workflow / production pipeline of Medellín is as follows. 

• The technical artists of Sequin AR used a tool called Brainstorm to execute this massive project. InfinitySet, Brainstorm’s VR and AR application takes data from live shooting and and send to Unreal engine in real time. Such tight integrated workflow gives perfect cohesion between real world and 3D world.
• To make it error free, the technical team worked in sync with dancing crew of Madonna and Maluma. It took care of other crucial parameters like lighting, shadows, reflections and many more.
• Exact time codes were designed for each dancing move and effects, so everything falls in place, at the right spot at the right time.
• Finally, the dance performance of Madonna’s Medellín was captured by three AR-enabled cameras: the crane / jib, a front-of-house camera and a wireless Steadicam. Wireless Steadicam was the best as it gave free camera movement.
• Everything was synced closely and streamed in real time to video screens. It broadcast AR enabled output which was combination of real characters, 3D volumetric data and visual effects.

Enjoy one of best AR experience, Madonna’s performance with her digital avatars of Medellín song: