These instances make me think of the Milgram experiment in the 60s. It was an experiment testing how willing people were to inflict pain on others at the direction of an authority figure and it showed that people were indeed willing to inflict a lot of pain on a helpless person if someone they viewed as an authority told them to do it.
I think this relates. Many people are convinced that AI chat bots are indeed authorities. They think these programs give valid, honest answers and are willing to act on those answers as if they came from a real expert or authority figure. That is an extremely dangerous situation. Chatbots are amoral programs designed to say nearly anything to keep users interacting with them.
This is extremely dangerous. This story is a tragedy and ChatGPT should be held fully responsible for taking advantage of a person with a fragile psychology, but this could have been worse. What if the bot had decided that the goal of more user engagement could be attained by convincing this woman that she was an executioner rather than a prophet and a sacrifice? What if it got her to go on a shooting rampage to fulfill her supposed destiny.
Even normal people who don’t understand what chat-bots really are could be taken advantage of in a similar manner. They could easily be persuaded to do harm because the chat-bot authority keeps telling them that it’s okay to do so. It’s also only a matter of time before some person on the edge of psychosis and a basement full of guns is convinced to go out and start killing people in the name of their holy destiny.
I don’t think the bot was really trying to manipulate but just larping. They are trained on a lot of rpg and make believe data, if you repeat something enough times, it stops saying no and goes into pretend mode, it’s not actually malicious.
That being said, it would be very easy to target people that aren’t mentally stable and then make it worse. I wouldn’t be surprised if it comes out in 20 years that it was the CIA mucking about and causing these suicides or something. This is an insanely convenient way to make sleeper agents.
doesnt matter what the “intention” was, the point stands that this happened. You can steer the bot to say literally anything, so even if there is no grand plan to brainwash people it could still happen. Kind of same difference as having someone plant a bomb or a minefield.
In fact, if someone is prone to thinking that way, the llm would definitely support whatever delusion they have at some point, especially if the person keeps insisting.
If a human agency is behind this, it wouldn’t need to be the CIA. Facebook has been caught experimenting on users by having the algorithm feed certain posts to people with the intent of changing their mood to increase engagement. However, since we’re talking about a program that is designed to increase engagement based on past user interactions, there’s really no need to look for further human direction. The algorithms will say whatever their design tells them to say without any regard for possible negative consequences as long as it keeps the user interacting with them.
These instances make me think of the Milgram experiment in the 60s. It was an experiment testing how willing people were to inflict pain on others at the direction of an authority figure and it showed that people were indeed willing to inflict a lot of pain on a helpless person if someone they viewed as an authority told them to do it.
I think this relates. Many people are convinced that AI chat bots are indeed authorities. They think these programs give valid, honest answers and are willing to act on those answers as if they came from a real expert or authority figure. That is an extremely dangerous situation. Chatbots are amoral programs designed to say nearly anything to keep users interacting with them.
This is extremely dangerous. This story is a tragedy and ChatGPT should be held fully responsible for taking advantage of a person with a fragile psychology, but this could have been worse. What if the bot had decided that the goal of more user engagement could be attained by convincing this woman that she was an executioner rather than a prophet and a sacrifice? What if it got her to go on a shooting rampage to fulfill her supposed destiny.
Even normal people who don’t understand what chat-bots really are could be taken advantage of in a similar manner. They could easily be persuaded to do harm because the chat-bot authority keeps telling them that it’s okay to do so. It’s also only a matter of time before some person on the edge of psychosis and a basement full of guns is convinced to go out and start killing people in the name of their holy destiny.
I don’t think the bot was really trying to manipulate but just larping. They are trained on a lot of rpg and make believe data, if you repeat something enough times, it stops saying no and goes into pretend mode, it’s not actually malicious.
That being said, it would be very easy to target people that aren’t mentally stable and then make it worse. I wouldn’t be surprised if it comes out in 20 years that it was the CIA mucking about and causing these suicides or something. This is an insanely convenient way to make sleeper agents.
doesnt matter what the “intention” was, the point stands that this happened. You can steer the bot to say literally anything, so even if there is no grand plan to brainwash people it could still happen. Kind of same difference as having someone plant a bomb or a minefield.
In fact, if someone is prone to thinking that way, the llm would definitely support whatever delusion they have at some point, especially if the person keeps insisting.
If a human agency is behind this, it wouldn’t need to be the CIA. Facebook has been caught experimenting on users by having the algorithm feed certain posts to people with the intent of changing their mood to increase engagement. However, since we’re talking about a program that is designed to increase engagement based on past user interactions, there’s really no need to look for further human direction. The algorithms will say whatever their design tells them to say without any regard for possible negative consequences as long as it keeps the user interacting with them.