Nvidia and Microsoft are solving a big problem with Copilot+

By Jacob Roach Published June 2, 2024

The Surface Laptop running local AI models. — Luke Larsen / Digital Trends

When Microsoft announced Copilot+ PCs a few weeks back, one question reigned supreme: Why can’t I just run these AI applications on my GPU? At Computex 2024, Nvidia finally provided an answer.

Nvidia and Microsoft are working together on an Application Programming Interface (API) that will allow developers to run their AI-accelerated apps on RTX graphics cards. This includes the various Small Language Models (SLMs) that are part of the Copilot runtime, which are used as the basis for features like Recall and Live Captions.

Recommended Videos

With the toolkit, developers can allow apps to run locally on your GPU instead of the NPU. This opens up the door to not only more powerful AI applications, as the AI capabilities of GPUs are generally higher than NPUs, but also the ability to run on PCs that don’t currently fall under the Copilot+ umbrella.

Get your weekly teardown of the tech behind PC gaming

It’s a great move. Copilot+ PCs currently require a Neural Processing Unit (NPU) that’s capable of at least 40 Tera Operations Per Second (TOPS). At the moment, only the Snapdragon X Elite satisfies that criteria. Despite that, GPUs have much higher AI processing capabilities, with even low-end models reaching to 100 TOPS, and higher-end options scaling even higher.

In addition to running on the GPU, the new API adds retrieval-augmented generation (RAG) capabilities to the Copilot runtime. RAG gives the AI model access to specific information locally, allowing it to provide more helpful solutions. We saw RAG on full display with Nvidia’s Chat with RTX earlier this year.

Performance comparison with the RTX AI toolkit. — Nvidia

Outside of the API, Nvidia announced the RTX AI Toolkit at Computex. This developer suite, arriving in June, combines various tools and SDKs that allow developers to tune AI models for specific applications. Nvidia says that by using the RTX AI Toolkit, developers can make models four times faster and three times smaller compared to using open-source solutions.

We’re seeing a wave of tools that enable developers to build specific AI applications for end users. Some of that is already showing up in Copilot+ PCs, but I suspect we’ll see far more AI applications at this point next year. We have the hardware to run these apps, after all; now we just need the software.

Topics

Lead Reporter, PC Hardware

Jacob Roach is the lead reporter for PC hardware at Digital Trends. In addition to covering the latest PC components, from…

Computing

Microsoft is finally making Copilot+ laptops useful for AI

Microsoft Surface Pro 11 front view in tablet mode.

Microsoft's Copilot+ PC initiative has been a smash hit, with many of them landing among the best laptops, but not for the AI hardware inside. Now, finally, Microsoft is putting the neural processing unit (NPU) inside Copilot+ PCs to good use. Starting today and going throughout the next two months, Microsoft will begin rolling out the 24H2 update for all Windows 11 PCs, and in the process, unlock several features for Copilot+ PCs, including the highly controversial Recall.

Recall is definitely the star of the show here, which will start showing up on Copilot+ laptops with a Snapdragon X chipset throughout October. Last week, Microsoft laid the groundwork for the release of Recall, detailing the security architecture of the feature and addressing some major criticisms of it. Now, for example, Recall is turned off instead of on by default. Microsoft is also allowing users to filter websites and giving users more control over their snapshots, including deleting them all.

Computing

Microsoft Copilot now has a voice and can ‘see what you see’ on the internet

Microsoft CEO Satya Nadella announces updates to the company's Copilot artificial intelligence (AI) tool.

You might want to start treating your web browser like you're always at work, at least if you want to use Microsoft's new Copilot Vision feature. The feature, which is natively built into Microsoft Edge, is able to "see what you see, and hear what you hear" as you navigate your browser, according to Microsoft's Executive Vice President Yusuf Mehdi.

All of this AI snooping isn't for nothing. Copilot Vision looks at what you're doing online to answer questions, provide recommendations, and summarize content. It can work with the new Copilot Voice feature, for example. Microsoft demoed the capabilities on Rotten Tomatoes, showing a user chatting with Copilot while browsing the website and looking for movie recommendations. Ultimately, Copilot settled on an Australian comedy for the Australian speaker, saying it made the choice because, "well, you're Australian." I guess that's taking personal context into account.

Computing

Microsoft outlines Recall security: ‘The user is always in control’

Recall promotional image.

Microsoft just released an update regarding the security and privacy protection in Recall. The blog post outlines the measures Microsoft is taking to prevent a data privacy disaster, including security architecture and technical controls. A lot of the features highlight that Recall is optional, and that's despite the fact that Microsoft recently confirmed that it cannot be uninstalled.

Microsoft's post is lengthy and covers just about every aspect of the security challenges that its new AI assistant has to face. One of the key design principles is that "the user is always in control." Users will be given the choice of whether they want to opt in and use Recall when setting up their new Copilot+ PC.

nproxy.org