Gemini’s ‘Screen Automation’ Can Order Food and Book Rides on Android
▼ Summary
– Google is developing a “screen automation” feature for Gemini to perform tasks like placing orders or booking rides within certain Android apps.
– This feature, found in the Google app 17.4 beta and codenamed “bonobo,” is built on groundwork laid by Android 16 QPR3.
– Google warns users that Gemini can make mistakes and that users are responsible for supervising its actions and can stop it at any time.
– For privacy, screenshots from app interactions may be reviewed by trained personnel to improve services if “Keep Activity” is enabled.
– The same beta also hints at a “Likeness” feature (codenamed “wasabi”) for creating 3D avatars, potentially for use in prompts.
Google is actively developing a powerful new feature for its Gemini AI assistant that will allow it to perform real-world tasks on your Android phone. This capability, known internally as “screen automation,” is designed to handle actions like ordering food and booking rides directly within certain apps. Found in the latest beta version of the Google app, this functionality hints at a future where your AI assistant can interact with your device’s screen to complete complex multi-step processes for you.
The feature, codenamed “bonobo,” is introduced with strings referencing “Get tasks done with Gemini.” It explains that the assistant can help with tasks by using screen automation on specific applications installed on a user’s device. This development builds upon groundwork laid in a recent Android update, indicating a deeper integration between the operating system and Google’s AI.
Google includes clear warnings about the experimental nature of this technology. The company states that Gemini can make mistakes and emphasizes that users are ultimately responsible for the assistant’s actions, advising close supervision. At any point, a user can stop the automated agent and manually take over a task. This underscores the importance of user control in these automated processes.
Privacy considerations are also addressed. When Gemini interacts with an app, screenshots may be reviewed by trained personnel to improve Google’s services, but only if a user’s “Keep Activity” setting is enabled. The company explicitly cautions against entering login credentials or payment information into Gemini chats for these automated tasks. It also advises avoiding the use of screen automation for emergencies or any activities involving sensitive personal data.
Separately, the same beta version contains references to another feature codenamed “wasabi.” This appears to relate to a “Likeness” integration, which is how the Android XR platform refers to the 3D avatars currently used in Google Meet. Code strings suggest users will be able to prepare and manage a digital likeness of themselves, complete with privacy assurances stating that the likeness can only be used by the individual who created it.
While these features discovered in the beta code are not guaranteed for public release, they provide a compelling glimpse into Google’s roadmap. The development of screen automation represents a significant step toward more proactive and capable AI assistants that can move beyond answering questions to actively performing useful tasks on a user’s behalf.
(Source: 9to5Google)
