The newest Windows Copilot agent can send emails, update documents on its own

19/10 20:15 - The newest Windows Copilot agent can send emails, update documents on its own
Microsoft’s Copilot will soon be able to work autonomously on local files in Windows, with Copilot Actions, an “experimental” feature that can perform tasks such as sending emails, updating documents and organizing files on its own.  The AI agent relies on a combination of AI vision and advanced reasoning to “click, type and scroll” like a human, Microsoft said. It follows the launch this past summer of a similar Copilot agent (currently available in preview) that can surf the web on its own, booking restaurants or planning trips on behalf of users, for instance.  Copilot Actions for local files, is “coming soon” to Windows Insiders in Copilot Labs, Microsoft.   Here’s how it works: users upload files to the Copilot chat interface in Windows, choose “take action” from a list of options, then describe the task they want completed. Microsoft cited an example of a user uploading photos and directing Copilot to fix the image orientation and remove duplicates. The agent will then perform the task in the background, interacting with relevant desktop and web applications to complete the job.  “Whether you need a hand sorting through your recent vacation photos or need to extract info from a PDF, Copilot Actions can do the heavy lifting for you based on the context of what’s on your PC,” said Yusuf Mehdi, Microsoft’s executive vice president and consumer chief marketing officer, said in a blog post Thursday.  While the feature is still in development, Copilot Actions has the potential to boost productivity by handling repetitive digital tasks, said Tom Mainelli, group vice president, device and consumer research, at IDC. “Copilot Actions is an ambitious step toward AI that can act on our behalf. It’s an important milestone that I’m eager to test myself,” he said. “Reliability, trust, and clear real-world value will ultimately determine how quickly it becomes mainstream.” Copilot Actions will detail each step as its performed and will request user approval in certain situations. Users also have the option to interrupt the agent and retake control at any point if they wish.  It remains to be seen just how reliable the agent will be and how comfortable users are having an agent interact with their files autonomously. Microsoft said it will start with a “narrow set of use cases” as the feature is tested with early users and the model is optimized prior to a full launch.  “You may see the agent make mistakes or encounter challenges with complex interfaces, which is why real-world testing of this experience is so critical to help us apply learnings to make this experience more capable and streamlined,” said Mehdi.  The company published a blog post outlining several measures to help users access Copilot Actions securely.  The tool will be disabled by default, for instance, and will have access to only a “limited set” of a user’s local folders — such as Documents, Downloads, Desktop, and Pictures — as well as other files accessible to all accounts on a system. “Only when the user provides authorization can Copilot Actions access data outside of these folders,” said Dana Huang, corporate vice president for Windows Security at Microsoft. Based on early users’ feedback, Microsoft said security and privacy controls will be further refined ahead of a full release. (The company came under fire with Recall — another AI feature for Windows, which takes regular screenshots of a user’s device — before making changes to it prior to launch.) Mainelli noted Microsoft’s efforts to address security concerns. “By isolating agents, limiting permissions, and giving users transparency into actions, the company is working to create a secure environment in this brave new world of agentic AI,” he said. “Of course, any agent that can act on local files introduces some new risks. I expect enterprise IT and security teams to thoroughly test Copilot Actions and validate these controls before large-scale deployment.” It’s important to note that Copilot Actions for Windows Copilot is different than Copilot Actions for the Microsoft 365 Copilot; the latter, announced late last year, is a lightweight workflow automation tool.  Microsoft made several other announcements today as it continues to embed AI features into Windows 11. It unveiled a text input option for Copilot Vision — a feature that can see a user’s screen and provide advice — to enable private interactions (coming to the Windows Insider program soon).  And it highlighted Copilot “connectors” that allow Windows Copilot to access files in Microsoft apps. These include OneDrive and Outlook, as well as Google services such as Google Drive, Gmail, Calendar, Contacts. The connectors allow users to surface files such as emails or PDFs directly from the chat interface so they can interact with them in different ways, such as creating summaries or checklists from the contents, for example.  ...