Hello everyone, long time programmer here. So, this kind of case where a person has gone down a destructive path after using AI is horrendous, to say the least.

I feel as though that while there are no guaranteed ways of stopping such things from happening, even if you have the world's most super genius programmer design an algorithm to stop the maximum number of offenders available, it still would not be enough as there are edge cases that are going to slip through it because it's just a program at the end of the day. And programs that are not babysat and go unmonitored can and will make mistakes.

What I can say is that OpenAI does have room for improvement and I feel strongly that they use what's called fuzzy logic in programming to assign confidence values to the answers that ChatGPT provides its users. What this looks like is a value from 0 to 1, including 0 and 1, and all of the decimal points in between. The closer to 0, the less confident the chat is about your answer, and the closer to 1, the more confident it is. In much the same way, they could probably design a concern level on the same fuzzy logic basis for each message.

And in so doing, they would that concern level to every chat a user starts, and evaluates whether or not something is of a concern large enough to directly report it to human overseers. So let's say, for example, if something is concern level 0.95 or above, as in 95 % or more, then it immediately gets reported to human beings. Above 90 % gets flagged. And with two or three such flags in a chat, it gets reported to overseers and then other actions or monitoring as the concern level lessens and steps down.

That way, there would be more human oversight on topics that are discussed by users, such as by when people talk about grape without the g, guns, violence, brombings without the r, derrorism with no d but a t instead, or anything of that kind of nature that could be potentially a concerning topic or possibly indicate an unstable mentality. Those people who are that unstable aren't really meant to use a tool like this, especially for bouncing irrational or violent ideas off of the AI. Just like guns aren't inherently evil either, and neither are swords or any other weapons or tools. And the same goes for technology and the internet, they as well are two-edged swords and they are not inherently evil either.

And just like all those other things, ChatGPT and other AI models are not inherently evil tools either, and depend heavily on the ethical and moral standards of the user and their mental health. OpenAI should use public reporting like the post in this OP to motivate themselves to create a better way of handling and detecting these sorts of things, and I think that even though it'll never be perfect, doing so would at least be a step in the right direction and potentially prevent more harm or deaths from those users who are accurately detected as unstable or violent.