Noted 6 decades ago: '...extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people'
We can’t say we weren’t warned.
From Joseph Weizenbaum, writing in 1976 (I think):
I had not realized…that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." Indeed, ELIZA’s code had not been designed to evoke this reaction in the first place. Upon observation, researchers discovered users unconsciously assuming ELIZA’s questions implied interest and emotional involvement in the topics discussed, even when they consciously knew that ELIZA did not simulate emotion
Weizenbaum was writing about one of the world’s first chatbots that he created in 1966, Eliza. It had the persona of a doctor and would correspond in text format with humans in the following manner:
Human: Well, my boyfriend made me come here.
ELIZA: Your boyfriend made you come here?
Human: He says I’m depressed much of the time.
ELIZA: I am sorry to hear you are depressed.
Human: It’s true. I’m unhappy.
ELIZA: Do you think coming here will help you not to be unhappy?
Basically rephrasing whatever the user said back at them as a question.
Nonetheless, he noticed it to be “surprisingly successful in eliciting emotional responses from users who, in the course of interacting with the program, began to ascribe understanding and motivation to the program’s output”. You can give reproductions of it a go on online now on sites like this and this.