The same technology and concept is what’s available on the new Galaxy S9. By collaborating with Samsung, ModiFace was able to optimize the experience.

The partnership could also kick start a new trend for phone manufacturers.

“Starting in house, we train our Neural Networks on thousands of images to create the base tracking and face analysis for our apps,” Houghton said. “This Neural Network is then embedded inside Samsung’s app. We worked with Samsung to tweak several parameters to make sure we were achieving the effects that brands and end users want to see.”

The partnership between the two companies could kick start a new trend for phone manufacturers. While beauty technology is still in its infancy, makeup has found its way on to our screens for years through social media. Whether it’s YouTube or Instagram tutorials, people are constantly looking to their smartphones to find new inspiration when it comes to makeup. With the Galaxy S9, all you need to do is swipe open the camera and you have a catalog of makeup at your disposal.

On the Galaxy S9, you can search through a variety of different cosmetic products available from both Sephora or Cover Girl, and you can purchase them on the spot. As you scroll through each product, it will apply itself to your face like Snapchat filters.

Using facial tracking and 3D video rendering, the makeup filters are mapped to the face at 30 frames per second. There’s no lag, and nothing is misaligned. It works as instantly as you to tap the next product you want to try on. You can try on complete looks – which includes lipstick, foundation, eyeshadow, blush, mascara and eyeliner – or you can try them all separately.

Trying on foundation can be extremely tricky, almost always requires being at the store and having an employee help you find your exact shade. There’s also different foundation types to take into account – matte, sheer, water-based, or ones that provide full coverage. To make sure the shades are as accurate as possible, the ModiFace team tediously went through each product and compared how it looked in real life versus how it looked on the smartphone. They worked with both Samsung and the brands directly to create a matching render.

“I myself was actually involved with that process and it’s kind of a lot of fun,” Houghton said. “You get to try on a lot of different things, and then you get to wipe them off and see how they look in the app. It involves a lot of screenshot comparison and manual work to make sure that the product we’ve included is just perfect.”

The team went through each product and compared how it looked in real life versus how it looked on the smartphone.

But there’s still always room for improvement. Using deep learning and image research, the team studies internal data — gathered from those who are testing the final product — to learn more about a user’s face in order to train its algorithms. This helps to produce better results when it comes to factors like how colors mix on to your lip, how light gets added into those colors, and how your face should look under different lighting if you have a specific foundation on. The data also helps to improve texture and overall coverage of the makeup.

With this kind of deep learning, the company is able to create new and more realistic effects. Recently, ModiFace partnered with L’Oreal to create 3D hair tracking that can recolor your hair in real time. The company has plans to work with Samsung to bring similar effects to its phones.

But for those with privacy concerns, Modiface doesn’t collect or monitor any data from its applications when in the hands of users. Any images you take with the apps will never leave your device.