Adobe’s AI Can Edit Entire Videos From a Single Frame
▼ Summary
– Adobe showcased experimental AI tools called “sneaks” at its Max conference for intuitive editing of photos, videos, and audio.
– Project Frame Forward enables video editors to add or remove objects across entire videos without masks, using AI to apply changes automatically.
– Project Light Touch uses generative AI to reshape light sources in photos, adjusting direction, diffusion, and color in real time.
– Project Clean Take allows users to modify speech delivery, emotion, or words in audio recordings while preserving the original voice characteristics.
– These sneaks are not publicly available and may not become official features, though some past sneaks have been integrated into Adobe’s software.
Adobe is pushing the boundaries of creative software with a suite of experimental AI tools previewed at its Max conference. These innovations, known internally as “sneaks,” promise to dramatically simplify complex editing tasks for photos, videos, and audio, offering more intuitive and powerful workflows for creators.
One of the standout demonstrations was Project Frame Forward, a tool designed to revolutionize video editing. This technology allows editors to add or remove elements from footage without the tedious process of manual masking. In a live demo, the AI identified and removed a person from the first frame of a video, seamlessly filling the space with a generated background. This single edit was then automatically propagated across the entire video clip, saving hours of manual labor. The tool also enables object insertion; users can simply draw a location and describe what they want to add via a text prompt. The AI not only generates the object but can also make it contextually aware, such as creating a puddle that accurately reflects the movement of an animal already present in the scene.
Another impressive sneak, Project Light Touch, uses generative AI to give photographers unprecedented control over lighting. It can alter the direction of light sources, simulate the effect of lamps that were off during the original shoot, and adjust the softness of shadows. Editors can even introduce dynamic lighting that can be dragged around the canvas, bending light realistically around subjects. This allows for creative effects like making a pumpkin appear to glow from within or transforming a daytime scene into a nighttime vista. The color temperature of these artificial lights is fully adjustable, enabling everything from warm, natural tones to vibrant, stylized RGB effects.
For audio post-production, Project Clean Take offers a powerful solution for fixing spoken dialogue. This tool can modify a speaker’s delivery or emotional tone, making them sound more cheerful or curious, for instance, without requiring a new recording session. It can even replace specific words while perfectly preserving the speaker’s unique vocal characteristics. The AI also excels at audio separation, isolating background noises into distinct tracks. This allows editors to mute or adjust individual sounds, such as a distracting hum, while maintaining the clarity and integrity of the primary voice recording.
The Max event featured several other notable experiments. Project Surface Swap enables instant changes to the material or texture of any object. Project Turn Style allows for the 3D-like rotation of objects within a 2D image, and Project New Depths treats photographs as 3D spaces, intelligently determining when newly inserted objects should be partially hidden by elements in the foreground.
It is important to note that these sneaks are not yet available to the public, and Adobe does not guarantee they will become official features in Creative Cloud or Firefly apps. However, the company has a strong track record of evolving its experimental projects into shipped products. Past sneaks have successfully transitioned into tools like Photoshop’s Distraction Removal and Harmonize, suggesting that some form of these groundbreaking AI capabilities will likely find their way into the hands of creatives in the future.
(Source: The Verge)
