DeepMind unveils Magic Pointer demos, coming to Gemini in Chrome
▼ Summary
– Google DeepMind developed the Magic Pointer to enable AI to understand not just what a user is pointing at, but why it is important to them.
– The AI pointer aims to replace text-heavy prompts with simpler interactions by capturing visual and semantic context around the cursor.
– Users can make complex requests in natural shorthand, such as pointing to a building image and saying “Show me directions.”
– Example use cases include pointing at a PDF for a bullet-point summary or hovering over statistics to request a pie chart.
– Google is rolling out the ability to use the pointer with Gemini in Chrome to interact with specific parts of a webpage, like comparing products.
The Magic Pointer, a new capability developed by Google DeepMind, is redefining how users interact with artificial intelligence by shifting the focus from text-heavy prompts to context-aware pointing. The research team behind this innovation explains that the goal is to create an AI that not only recognizes what the pointer is indicating but also understands why that object matters to the user.
A major frustration with current AI tools is that they operate in isolated windows, forcing users to drag information into them. DeepMind aims to flip this dynamic entirely. The vision is for AI to be intuitively present across all applications, seamlessly integrating into the user’s workflow without interrupting it. For instance, pointing at an image of a building and simply saying “Show me directions” should be enough. The AI, already aware of the context, would handle the rest.
This approach replaces complex, text-heavy prompts with simpler, more natural interactions. An AI-enabled pointer can smoothly capture both the visual and semantic context around the cursor, allowing the computer to “see” and comprehend what the user considers important.
By combining context, pointing, and speech, the AI system can interpret complex requests delivered in natural shorthand. Practical examples include pointing at a PDF and asking for a bullet-point summary to paste into an email, hovering over a table of statistics to request a pie chart version, or highlighting a recipe to have all ingredients doubled.
In one demonstration, a paused frame in a travel video instantly becomes a booking link for a restaurant shown on screen. Google has already made two AI-enabled pointer demos available in AI Studio.
Furthermore, users will soon be able to use their pointer to ask Gemini in Chrome about specific parts of a webpage. This feature is currently rolling out. For example, you can select several products on a page and ask the AI to compare them, or point to a spot in your living room photo to visualize a new couch there.
(Source: 9to5google.com)
