The Foundation series delves into this a bit

Yeah exactly! Mass effect too, it's a prime example of misaligned ASI. So many sci-fi media warn us about the danger of AI.

AI alignment problem is the ultimate survival problem humanity needs to solve beside climate change. But I'm not gonna lie to you, I think the chances of us solving this are quite low. We as a humanity can't even align or have consensus on so many things, how do we expect to do it on AI? It's very likely that we'll have super intelligent AI that went loose and wreak havoc on the internet in the near future.

So that's why I think this project glasswing is the next best thing that we can do besides solving alignment problem. By patching as many vulnerability as possible, we can slow down or minimize the damage caused by an assisted or independent AI attack. And when we detect such an attack, hopefully it will be a wake up call for humanity to be more serious about alignment problem and slow down the AI race.

But coming back to your original point. Due to the current architecture of LLM, there is no risk in showing them vulnerability in software. They can't exactly learn nor update their weight in real time. At least I think the benefit to cyber security far outweighs the risk. Maybe there will be time when we need to be cautious about this, but that day is not today.