OpenAI Aims to Make ChatGPT a Universal App Interface
▼ Summary
– OpenAI launched an SDK preview allowing developers to build apps inside ChatGPT with full control over data, actions, and user interfaces.
– The SDK is built on the open source Model Context Protocol (MCP), simplifying integration for existing MCP users by adding an HTML resource.
– ChatGPT can now directly interact with apps like Figma to perform tasks such as converting sketches into diagrams within the conversation window.
– The new API enables ChatGPT to access and respond to real-time app context, such as explaining content from an embedded Coursera video.
– Demos showed ChatGPT using multiple apps simultaneously, like generating Canva posters while users interact with a Zillow map, maintaining chat functionality even in full-screen mode.
During a recent keynote event, OpenAI revealed its ambitious plan to transform ChatGPT into a versatile platform capable of hosting and running third-party applications directly within its interface. This new Software Development Kit (SDK) preview grants developers comprehensive control over app data, action triggers, and interactive user interfaces, all seamlessly integrated into ongoing ChatGPT conversations. The move signals a significant step toward making the AI assistant a central hub for digital tasks and services.
Built upon the open source Model Context Protocol (MCP), the SDK simplifies integration for developers already familiar with the framework. They can enable ChatGPT compatibility by simply adding an HTML resource, streamlining the process of bringing external applications into the conversational AI environment. This approach lowers the technical barrier for creating deeply integrated, interactive experiences.
Practical applications of this technology were demonstrated live, showcasing how users can now accomplish complex multi-step tasks without leaving the chat. For example, a user could ask Figma to convert a rough sketch into a polished diagram and view the results directly within their ChatGPT window. Similarly, when requesting music recommendations, ChatGPT could not only suggest songs but also automatically generate and present a customized Spotify playlist.
OpenAI engineer Alexi Christakis illustrated how the system allows applications to share contextual information with ChatGPT in real time. He described this as ChatGPT effectively “talking to apps,” enabling the AI to provide instant commentary and assistance based on live application content. During his demonstration, ChatGPT continuously analyzed and explained the content of an embedded Coursera video without requiring any manual input about what was happening on screen.
Additional demonstrations highlighted the platform’s multitasking capabilities. While a user consulted an inline Zillow map for property information, ChatGPT simultaneously worked with Canva in the background to generate poster design ideas. Notably, even when the Zillow interface expanded to full screen, users maintained the ability to query ChatGPT through an overlaid chat window, preserving access to AI assistance regardless of which application had focus.
(Source: Ars Technica)
