Over the decades, VFX has been an integral part of movie making and storytelling. But, creating realistic and seamless visual effects shots are a time-consuming and labor-intensive process that requires skilled artists and advanced software tools. In today’s advanced technology age of AI (Artificial Intelligence), now there’s a new tool called CopyCat, a Machine Learning tool of Nuke that helps them work faster and smarter. It is developed by the parent company Foundry for Nuke, its flagship compositing software.
As the name suggests, its uses AI techniques that learns from the artist and takes care of some of the mundane tasks. So, he/she can focus more on the creative stuff. Let’s understand all these in detailed manner.
What is CopyCat?
As mentioned earlier, CopyCat is a one of the ML tool of Nuke, the leading VFX Compositing software worldwide. It enables artists to train neural networks for various visual effects tasks.
It was introduced in Nuke 13.0 version. The core focus was to streamline the VFX workflow and pipeline using ML toolsets. It is the need of hour as the project deadlines are getting smaller and production budgets are being tighter. Artists needs a fresh approach to deliver shots on time. The AVGC industry needs some AI based solutions to tackle this situation.
CopyCat is Foundry’s answer to this scenario. It is a Nuke plug-in, which ships with the bundle. This AI tool for VFX is a time savior when an artist has to deal with complex / time consuming sequences.
How CopyCat works?
The process of machine learning tool CopyCat is as follows.
- It all happens through the techniques of Machine Learning (ML) algorithm.
- The VFX artist creates a VFX effect (roto, paint, blur, defocus etc.) on some frames.
- These are termed as ‘reference frames’ or ‘data sets’. These are feed to CopyCat plugin.
- It train the neural networks to execute the machine learning process.
- These neural networks are supplied to the ‘Inference’ node.
- The ‘Inference’ node replicate the ‘learned data’ across entire sequences or shots.
This AI based prototype can be tweaked as per the requirements. So, it gives a much needed flexibility and creativity to the artist.
In a nutshell, you can tailor machine learning and its algorithms as per the needs of your project. CopyCat provides a user-friendly interface and graphical monitoring system for ease of use.
Having said all these, please don’t consider this Machine Learning Nuke tool as a magic wand. The solution may take time depending on various factors such as frame size, complexity of shot (lighting, grains, fast movements etc.), hardware configuration and many others. But, if you continue to update the neural network training, the results will come close to perfection.
Apart from CopyCat and Inference, there are two more nodes in the Nuke’s machine learning tool kit.
- Upscale: You can the resize the footage by x2.
- Deblur: It can remove motion blur from footage.
This machine learning VFX toolset is immensely powerful for your post production workflow and pipeline. It gives a creative freedom to the artist, so he/she can focus of the more challenging aspects of the visual effects shot and CopyCat can work in background to take care of the mundane works.
Best part of all these Nuke ML plugins are that they are open ended. So, any VFX artist can incorporate it in their post production pipeline and reduce repetitive works. However, the result and success depends on the quality and quantity of the input data provided by the artist. It may not be 100% accurate, but it certainly does the close work.
Use cases / Benefits of CopyCat – the Machine Learning tool of Nuke
As a ML tool, it focuses on reducing the work of repetitious works. Some of the major use cases and benefits of CopyCat are as follows.
1. Matte generation
One of the key advantages of CopyCat is its ability to reduce the time and effort required for rotoscoping, a tedious and time-consuming task that involves manually separating foreground and background elements in a shot. It can create the rotoscoping automation process for an efficient VFX post production pipeline.
This Foundry machine learning tool can automate rotoscoping tasks by training neural networks to generate garbage mattes (to mask out unwanted elements from a shot) based on reference frames. The reference frames contain the desired foreground and background elements. Based on these inputs, the machine learning algorithm generates garbage mattes for the rest of the shot.
This streamlines the process, saving artists valuable time and enabling them to focus on more creative aspects of their work. It is a great Nuke plug in for rotoscopy automation.
2. Beauty work
Another use case for CopyCat is beauty work, which involves enhancing or correcting the appearance of actors or objects in a shot. This can include removing blemishes, wrinkles, or other imperfections, as well as adjusting skin tone, hair color, or other attributes. With CopyCat, artists can train a neural network to automatically apply beauty work to a shot based on a small number of reference frames.
The toolset allows artists to adjust the level of beauty work applied and preview the results in real-time, enabling them to achieve the desired look quickly and efficiently. It improves the overall quality of the visual effects sequence.
3. De-aging
De-aging actors in shots can be a complex task, but CopyCat simplifies this process by leveraging machine learning capabilities through running various algorithms. It involves making an actor appear younger or older than their actual age, which can be challenging to achieve convincingly. The tool allows artists to adjust the level of de-aging applied and preview the results in real-time, enabling them to achieve the desired look quickly and efficiently.
Challenges of using CopyCat and neural network training in VFX workflow
Well, as said earlier, it is an AI and ML based VFX tool. It relies on machine learning algorithms to analyze and learn from the input data, so if the data is noisy, incomplete, or biased, the results may not be accurate or realistic. Therefore, it is important for artists to provide high-quality and diverse input data to achieve the best results.
In an essence, it works on data. The more relevant and high quality data you feed in, the more accurately it can work.
It’s a process, it cannot be achieved overnight. Foundry is working on it continuously to overcome limitations and future developments of CopyCat to make it better day by day. It is set to become a game changer in the AVGC industry.
Conclusion
CopyCat is a powerful machine learning tool of Nuke that can help artists achieve more efficient and creative VFX workflows. Its ability to reduce the time and effort required for tasks such as rotoscoping, beauty work, and de-aging can save artists hours of manual labor. It has been praised by many early adapter visual effects artists and studios for its performance.
Nevertheless, CopyCat is a promising toolset that has the potential to revolutionize the VFX industry and enable artists to achieve more realistic and seamless visual effects shots.