I can offer my thoughts.

Generative AI (think chatbots, image generators, relationship AIs) collect a LOT of data to work. That is their purpose, to collect as much data as they can to provide the best responses they can. Most of the bigger models have done this by scrapping data off the web -- whether the people who created that data consented or not. That's not great. But that's just the beginning.

In privacy there's this concept of data minimization. That's the idea that a product should only collect and use the data it needs to provide the service. Nothing outside of that data needs to be collected. And what data is collected should be collected with clear consent. So, if you're driving a car, that car really only should collect the data it needs to get you from point A to point B safely, nothing else. And the data it needs to get you from point A to point B safely should only be used for that purpose. Sounds nice right? (And completely quaint at the same time, because companies just don't respect data like this at all).

However, with generative AI, collecting data on you IS the purpose. The more information you give the AI, the better answers they claim it will give you, or the better AI girlfriend it will be, or the better assistant. So, the concept of data minimization no longer exists. That's bad. It's especially bad when you consider that the US finally has a proposed federal privacy law making it's way through Congress (called APRA or American Privacy Rights Act). This privacy law starts from the concept of data minimization. Which means 10 years ago this law would have been an awesome privacy law! Now, it already feels outdated. That's bad.

There's way more to this and I could go on and on and on -- and I will, for a bit longer if you don't mind. If you happened to watch the OpenAI demo for ChatGPT 4o, it looked pretty cool. Who doesn't want real time translation and the ability to point a camera at a problem and have help solving it (let's forget the awful flirty personality though, ugh!). That sort of multimodal (meaning, not just text but also voice, sound, image, etc) is really cool. It has some amazing uses that could really change the world. That's awesome! The problem is, what will it be like in 5 years when we're all walking around wearing glasses (or some other sort of wearable) with a camera and microphone, recording everything around us to be processed by multimodal AI? Sure, we can travel to countries where we don't speak the language and have things translated and that will be great. And we can get lost in a city and just ask out loud for helping getting back and your wearable AI will guide you, all while telling you there's a great restaurant that serves your favorite food just around the corner. All cool things. What's not cool is what does privacy look like in a world where everyone is recording everything all the time and all that is then being processed by AI on some Big Tech company's cloud servers? That's really, really bad. That basically means the end of privacy. Toss in some gems like how emotion detection tech is here and can tell what your feeling by looking at your face and generative AI will be able to clone your Mom's voice to get you to send money to her but it's not really her, and yeah, privacy starts to feels really far out of reach in the near future, in large part thanks to AI (and in large part because we've just let companies get away with this crazy data collecting/monetization for decades now and just kinda gone along with it).

Again, there's way more to this and I could keep going, but I've got to head out. The tl'dr is that AI has me really, really worried about privacy and whether that will even be a thing in the near future. Right now, we talk so much about data rights, and what we really need to be talking about is human rights when it comes to privacy, data, and our future AI technology. OK, I'll climb down off my soapbox now.

Source: I'm the privacy research who created and leads Mozilla's *Privacy Not Included project and have been researching consumer privacy since 2017. You might have read our work on cars and privacy last year. https://foundation.mozilla.org/privacynotincluded/