Deep Mind showed off the latest results from its generative AI video-to-audio research on Tuesday. It’s a novel system that combines what it sees on-screen with the user’s written prompt to create synced audio soundscapes for a given video clip.
The V2A AI can be paired with vide -generation models like Veo, Deep Mind’s generative audio team wrote in a blog post, and can create soundtracks, sound effects, and even dialogue for the on-screen action. What’s more, Deep Mind claims that its new system can generate “an unlimited number of soundtracks for any video input” by tuning the model with positive and negative prompts that encourage or discourage the use of a particular sound, respectively.
The system works by first encoding and compressing the video input, which the diffusion model then leverages to iteratively refine the desired audio effects from background noise based on the user’s optional text prompt and from the visual input. This audio output is finally decoded and exported as a waveform that can then be recombined with the video input.
The best part is that the user doesn’t have to go in and manually (read: tediously) sync the audio and video tracks, as the V2A system does it automatically. “By training on video, audio and the additional annotations, our technology learns to associate specific audio events with various visual scenes, while responding to the information provided in the annotations or transcripts,” the Deep Mind team wrote.
The system is not yet perfected, however. For one, the output audio quality is dependent on the fidelity of the video input and the system gets tripped up when video artifacts or other distortions are present in the input. According to the Deep Mind team, syncing dialogue to the audio track remains an ongoing challenge.
“V2A attempts to generate speech from the input transcripts and synchronize it with characters’ lip movements,” the team explained. “But the paired vide- generation model may not be conditioned on transcripts. This creates a mismatch, often resulting in uncanny lip-syncing, as the video model doesn’t generate mouth movements that match the transcript.”
The system still needs to undergo “rigorous safety assessments and testing” before the team will consider releasing it to the public. Every video and soundtrack generated by this system will be affixed with Deep Mind’s SynthID watermarks. This system is far from the only audio-generating AI currently on the market. Stability AI dropped a similar product just last week while ElevenLabs released their sound effects tool last month.
Mozilla is giving Firefox’s Smart Window two new tricks. If you haven’t tried it yet, Smart Window is an optional AI mode in Firefox with a built-in chat that helps you get things done while you browse. With this update, it can fill out online forms using details from your tabs and keep an eye on websites for you.
Tired of typing the same details into every form?
The absolutely absurd idea of putting a touchscreen display on a mouse isn't exactly new. Turtle Beach already did it with its Command Series MC7, which features a 2.25-inch control screen. But apparently, even that wasn't ridiculous enough for Arctic Fox Technologies. Its new Super Mouse looks like someone took the chunky old Atari ST mouse, slapped a touchscreen and scroll wheel on it, and called it an upgrade.
The result is a rather unusual-looking mouse that is now on Kickstarter, starting at $199. And as ridiculous as it looks, some of its features are genuinely intriguing.
Apple is less than a day away from launching its highly anticipated OLED MacBook Pro with a touch-sensitive screen. But it appears that the upgrade is going to cost a pretty penny. According to a report by Bloomberg, Apple is reportedly eyeing a starting price of $2,899 for the 14-inch MacBook Pro, while the 16-inch version could start at $3,399 in the US.
"At $2,899, the 14-inch model would mark the highest starting price for a new MacBook Pro model, surpassing the $2,799 Apple charged for a 17-inch version introduced in 2006," says the report. The report claims that Apple is targeting a minimum price hike of $400 for its upcoming redesigned 14-inch and 16-inch MacBook Pro models.