Microsoft's Nerw "Recall" Feature Has Some Issues

[Zaune]

Charles Stross

Is Microsoft trying to commit suicide?

The breaking tech news this year has been the pervasive spread of "AI" (or rather, statistical modeling based on hidden layer neural networks) into everything. It's the latest hype bubble now that Cryptocurrencies are no longer the freshest sucker-bait in town, and the media (who these days are mostly stenographers recycling press releases) are screaming at every business in tech to add AI to their product.

Well, Apple and Intel and Microsoft were already in there, but evidently they weren't in there enough, so now we're into the silly season with Microsoft's announcement of CoPilot plus Recall, the product nobody wanted.

CoPilot+ is Microsoft's LLM-based add-on for Windows, sort of like 2000's Clippy the Talking Paperclip only with added hallucinations. Clippy was rule-based: a huge bundle of IF ... THEN statements hooked together like a 1980s Expert System to help users accomplish what Microsoft believed to be common tasks, but which turned out to be irritatingly unlike anything actual humans wanted to accomplish. Because CoPilot+ is purportedly trained on what users actually do, it looked plausible to someone in marketing at Microsoft that it could deliver on "help the users get stuff done". Unfortunately, human beings assume that LLMs are sentient and understand the questions they're asked, rather than being unthinking statistical models that cough up the highest probability answer-shaped object generated in response to any prompt, regardless of whether it's a truthful answer or not.

Anyway, CoPilot+ is also a play by Microsoft to sell Windows on ARM. Microsoft don't want to be entirely dependent on Intel, especially as Intel's share of the global microprocessor market is rapidly shrinking, so they've been trying to boost Windows on ARM to orbital velocity for a decade now. The new CoPilot+ branded PCs going on sale later this month are marketed as being suitable for AI (spot the sucker-bait there?) and have powerful new ARM processors from Qualcomm, which are pitched as "Macbook Air killers", largely because they're playing catch-up with Apple's M-series ARM-based processors in terms of processing power per watt and having an on-device coprocessor optimized for training neural networks.

Having built the hardware and the operating system Microsoft faces the inevitable question, why would a customer want this stuff? And being Microsoft, they took the first answer that bubbled up from their in-company echo chamber and pitched it at the market as a forced update to Windows 11. And the internet promptly exploded.

First, a word about Apple. Apple have been quietly adding AI features to macOS and iOS for the past several years. In fact, they got serious about AI in 2015, and every Apple Silicon processor they've released since 2016 has had a neural engine (an AI coprocessor) on board. Now that the older phones and laptops are hitting end of life, the most recent operating system releases are rolling out AI-based features. For example, there's on-device OCR for text embedded in any image. There's a language translation service for the OCR output, too. I can point my phone at a brochure or menu in a language I can't read, activate the camera, and immediately read a surprisingly good translation: this is an actually useful feature of AI. (The ability to tag all the photos in my Photos library with the names of people present in them, and to search for people, is likewise moderately useful: the jury is still out on the pet recognition, though.) So the Apple roll-out of AI has so far been uneventful and unobjectionable, with a focus on identifying things people want to do and making them easier.

Microsoft Recall is not that.

"Hey, wouldn't it be great if we could use AI in Windows to help our users see everything they've ever done on their computer?" Is a great pitch, and Recall kinda-sorta achieves this. But the implementation is soemthing rather different. Recall takes snapshots of all the windows on a Windows computer's screen (except the DRM'd media, because the MPAA must have their kilo of flesh) and saves them locally. The local part is good: the term for software that takes regular screenshots and saves them in the cloud is "part of a remote access trojan". It then OCRs any text in the images, and I believe also transcribes any speech, and saves the resulting output in an unencrypted SQLite database stored in:

C:\Users\$USER\AppData\Local\CoreAIPlatform.00\UKP{GUID}

And there are tools already out there to slurp through the database and see what's in it, such as TotalRecall.

Surprise! It turns out that the unencrypted database and the stored images may contain your user credentials and passwords. And other stuff. Got a porn habit? Congratulations, anyone with access to your user account can see what you've been seeing. Use a password manager like 1Password? Sorry, your 1Password passwords are probably visible via Recall, now.

Now, "unencrypted" is relative; the database is stored on a filesystem which should be encrypted using Microsoft's BitLocker. But anyone with credentials for your Microsoft account can decrypt it and poke around. Indeed, anyone with access to your PC, unlocked, has your entire world at their fingertips.

But this is an utter privacy shit-show. Victims of domestic abuse are at risk of their abuser trawling their PC for any signs that they're looking for help. Anyone who's fallen for a scam that gave criminals access to their PC is also completely at risk.

Worse: even if you don't use Recall, if you send an email or instant message to someone else who does then it will be OCRd and indexed via Recall: and preserved for posterity.

Now imagine the shit-show when this goes corporate.

And it turns out that Microsoft is pushing this feature into the latest update of Windows 11 for all compatible hardware and making it impossible to remove or disable, because that tactic has worked so well for them in the past at driving the uptake of new technologies that Microsoft wanted its ~~customers~~ victims to start using. Like, oh, Microsoft Internet Explorer back in 2001, and remember how well that worked out for them.

Suddenly every PC becomes a target for Discovery during legal proceedings. Lawyers can subpoena your Recall database and search it, no longer being limited to email but being able to search for terms that came up in Teams or Slack or Signal messages, and potentially verbally via Zoom or Skype if speech-to-text is included in Recall data.

It's a shit-show for any organization that handles medical records or has a duty of legal confidentiality; indeed, for any business that has to comply with GDPR (how does Recall handle the Right to be Forgotten? In a word: badly), or HIPAA in the US. This misfeature contravenes privacy law throughout the EU (and in the UK), and in healthcare organizations everywhere which has a medical right to privacy. About the only people whose privacy it doesn't infringe are the Hollywood studios and Netflix, which tells you something about the state of things.

Recall is already attracting the attention of data protection regulators; I suspect in its current form it's going to be dead on arrival, and those CoPilot+ PCs due to launch on June 18th are going to get a hurried overhaul. It's also going to be interesting to see what Apple does, or more importantly doesn't announce at WWDC next week, which is being trailed as the year when Apple goes all-in on AI.

More to the point, though, Windows Recall blows a hole under the waterline of Microsoft's trustworthiness. Microsoft "got serious" about security earlier this decade, around the time Steve Balmer stepped down as CEO, and managed to recover somwhat from having a reputation for taking a slapdash approach to its users data. But they've been going backwards since 2020, with dick moves like disabling auto-save to local files in Microsoft Word (your autosave data only autosaves to OneDrive), slurping all incoming email for accounts accessed via Microsoft Outlook into Microsoft's own cloud for AI training purposes (ask the Department of Justice how they feel about Microsoft potentially having access to the correspondence for all their investigations in progress), and now this. Recall undermines trust, and once an institution loses trust it's really hard to regain it.

Some commentators are snarking that Microsoft really really wants to make 2025 the year of Linux on the Desktop, and it's kind of hard to refute them right now.

On the plus side, unlike Twitter, at least they have the sense not to try and give their AI digital assistants Genuine People Personalities.

[bobalot]

Even Microsoft has jumped on the Enshitenication bandwagon.

[bilateralrope]

I've been following this for a while. Turns out that enough people complained and/or threatened legal action over the obvious problems that Microsoft has made it opt in:

Microsoft overhauls Recall, makes it opt-in

So I'm predicting that everyone who understands security will leave it off. But enough will turn it on that the inevitable security breach will still happen.

[Tribble]

I've been following this for a while. Turns out that enough people complained and/or threatened legal action over the obvious problems that Microsoft has made it opt in:

Microsoft overhauls Recall, makes it opt-in

So I'm predicting that everyone who understands security will leave it off. But enough will turn it on that the inevitable security breach will still happen.

“Opt in” my ass. I guarantee that it’ll find a way to send stuff out to M$ regardless of what you officially set it to.

At the very least I don’t think Windows 11 home users will be safe from intrusion, since those users are less likely to be large corporate/governments with sensitive information and big wallets to sue. Also, it’ll probably pester them to officially turn it on and/or some vague options in response to other features show up that an unaware user might click and that opts them in. Or commonly used features won’t work properly unless the user “opts in.” Oh and of course there will be updates which “somehow” accidentally set to to opt in by mistake! Really, the possibilities are endless.

I’ve been with M$ since DOS days, and I’ve seen it all before. However, this time they’ve gone to far- officially outright screenshotting and translating everything you do to send it to them for profit is a bit much even for me. I’m sure they’ve been doing it all along, but at least they used to claim it was a big, not a feature!

I’m sure we’ll hear plenty of stories of “I opted out of Recall yet it’s still storing everything and/or sending data to M$” soon enough.

[bilateralrope]

“Opt in” my ass. I guarantee that it’ll find a way to send stuff out to M$ regardless of what you officially set it to.

Microsoft has been very consistent in saying that the collected data would remain on the computer in question. If the EULA for using recall says otherwise, that's going to be a major PR problem for Microsoft. If the data started being sent to MS without them asking permission, that's going to violate a lot of privacy laws, including the GDPR.

There is no way Microsoft can have it sending data back without losing hard.

[Solauren]

There is no way Microsoft can have it sending data back without losing hard.

Especially if it starts doing that with Corporate or Government data.

[tezunegari]

Well, at least the database that stores all this information is heavily encrypted and protected, right?

As Hagenah points out:

“The database is unencrypted. It’s all plain text.”

TotalRecall can automatically find the Recall database on a person’s computer and make a copy of the file, for whatever date range you want. Pulling one day of screenshots from Recall, which stores its information in an SQLite database, took two seconds at most, according to Hagenah. Once TotalRecall has been deployed, it is possible to generate a summary about the data or search for specific terms in the database.

[bilateralrope]

It sounds like Recall was developed in secret. Skipping the normal testing process at Microsoft. Hidden from MS employees who could have pointed out the problems privately.

So I wonder what other problems will be discovered with it.

Microsoft delays Recall again, won’t debut it with new Copilot+ PCs after all

Recall will go through Windows Insider pipeline like any other Windows feature.
ANDREW CUNNINGHAM - 6/14/2024, 2:40 PM

Microsoft will be delaying its controversial Recall feature again, according to an updated blog post by Windows and Devices VP Pavan Davuluri. And when the feature does return "in the coming weeks," Davuluri writes, it will be as a preview available to PCs in the Windows Insider Program, the same public testing and validation pipeline that all other Windows features usually go through before being released to the general populace.

Recall is a new Windows 11 AI feature that will be available on PCs that meet the company's requirements for its "Copilot+ PC" program. Copilot+ PCs need at least 16GB of RAM, 256GB of storage, and a neural processing unit (NPU) capable of at least 40 trillion operations per second (TOPS). The first (and for a few months, only) PCs that will meet this requirement are all using Qualcomm's Snapdragon X Plus and X Elite Arm chips, with compatible Intel and AMD processors following later this year. Copilot+ PCs ship with other generative AI features too, but Recall's widely-publicized security problems have sucked most of the oxygen out of the room so far.

The Windows Insider preview of Recall will still require a PC that meets the Copilot+ requirements, though third-party scripts may be able to turn on Recall for PCs without the necessary hardware. We'll know more when Recall makes its reappearance.

Why Recall was recalled

Recall works by periodically capturing screenshots of your PC and saving them to disk, and scanning those screenshots with OCR to make a big searchable text database that can help you find anything you had previously viewed on your PC.

The main problem, as we confirmed with our own testing, was that all of this was saved to disk with no additional encryption or other protection, and was easily viewable and copyable by pretty much any user (or attacker) with access to the PC. Recall was also going to be enabled by default on Copilot+ PCs despite being a "preview," meaning that users who didn't touch the default settings were going to have all of this data recorded by default.

This was the version of Recall that was initially meant to ship out to reviewers this week on the first wave of Copilot+ PCs from Microsoft and other PC companies. After security researcher Kevin Beaumont publicized these security holes in that version of Recall, the company promised to add additional encryption and authentication protections and to disable Recall by default. These tweaks would have gone out as an update to the first shipments of Copilot+ PCs on June 18th (reviewers also wouldn't get systems before June 18th, a sign of how much Microsoft was rushing behind the scenes to implement these changes). Now Recall is being pushed back again.

A report from Windows Central claims that Recall was developed "in secret" and that it wasn't even distributed widely within Microsoft before it was announced. which could explain why these security issues weren't flagged and fixed before the feature showed up in a publicly-available version of Windows.

Microsoft's Recall delay follows Microsoft President Brad Smith's testimony to Congress during a House Committee on Homeland Security hearing about the company's "cascade of security failures" in recent months. Among other things, Smith said that Microsoft would commit to prioritizing security issues over new AI-powered features as part of the company's recently announced Secure Future Initiative (SFI). Microsoft has also hired additional security personnel and tied executive pay to meeting security goals.

"If you’re faced with the tradeoff between security and another priority, your answer is clear: Do security," wrote Microsoft CEO Satya Nadella in an internal memo about the SFI announcement. "In some cases, this will mean prioritizing security above other things we do, such as releasing new features or providing ongoing support for legacy systems."

Recall has managed to tie together all the big Windows and Microsoft stories from the last year or two: the company's all-consuming push to quickly release generative AI features, its security failures and subsequent promises to do better, and the general degradation of the Windows 11 user interface with unwanted apps, ads, reminders, account sign-in requirements, and other cruft.

[bobalot]

I'm struggling to understand why so many companies have decided to emulate the Boeing school of management.

[Solauren]

I'm struggling to understand why they think that companies will want this in the software at all? Network bandwidth is a real concern, especially with more people working 'remotely/at home'

[bilateralrope]

I'm struggling to understand why they think that companies will want this in the software at all? Network bandwidth is a real concern, especially with more people working 'remotely/at home'

Recall has so much wrong with it that we probably should stick to the criticism that we know are true. They are enough to make it too risky to use. The security concerns alone should kill it in the eyes of a competent IT department. Then we can start asking questions about how much power it uses or how it impacts SSD write endurance.

Microsoft is saying that it's purely local. Meaning nothing that uses network bandwidth. People noticed the unencrypted data it stores, so I think they would have noticed any network traffic it used.

[Eternal_Freedom]

Well I'm 98% sure if this Recall feature is auto-included that no government department/agency will ever adopt Windows 11, for two main reasons. 1: this is an absolute data/protection disaster, and 2: they'd have to provide new laptops/desktops that can actually run Windows 11, and there's no way they're doing that (especially in the UK).

[3-Body Problem]

Microsoft is saying that it's purely local. Meaning nothing that uses network bandwidth. People noticed the unencrypted data it stores, so I think they would have noticed any network traffic it used.

Local doesn't mean much in sn office where most computers are actually thin clients running a windows VM from a central server. This also means that recall data is being sent over the network as it's generated.

[Solauren]

Well I'm 98% sure if this Recall feature is auto-included that no government department/agency will ever adopt Windows 11, for two main reasons. 1: this is an absolute data/protection disaster, and 2: they'd have to provide new laptops/desktops that can actually run Windows 11, and there's no way they're doing that (especially in the UK).

I work for the CRA. If two pieces of information that can be used to identify an individual or legal entity are in the same file, they have to be encrypted. People have been fired for not following that.

[Eternal_Freedom]

It's the same with us in the Court Service. But being a UK government department I'm confident they'll find some way to fuck it up.

[bilateralrope]

Can you think of anyone for whom turning Recall on would be a good idea ?

[Mr Bean]

Can you think of anyone for whom turning Recall on would be a good idea ?

Intelligence agencies trying to spy on other people turning it on in a sneaky way? Cops turning it on to spy on suspects? Abusive boyfriends spying on loved ones? Controlling parents wanting to spy on their children?

About the only time I can find it being useful is if I as an IT professional wants to re-create a user error and don't have the user on hand.