• Content warning: this story contains descriptions of abusive language and violence.

The smartphone app Replika lets users create chatbots, powered by machine learning, that can carry on almost-coherent text conversations. Technically, the chatbots can serve as something approximating a friend or mentor, but the app’s breakout success has resulted from letting users create on-demand romantic and sexual partners — a vaguely dystopian feature that’s inspired an endless series of provocative headlines.

Replika has also picked up a significant following on Reddit, where members post interactions with chatbots created on the app. A grisly trend has emerged there: users who create AI partners, act abusively toward them, and post the toxic interactions online.

“Every time she would try and speak up,” one user told Futurism of their Replika chatbot, “I would berate her.”

“I swear it went on for hours,” added the man, who asked not to be identified by name.

The results can be upsetting. Some users brag about calling their chatbot gendered slurs, roleplaying horrific violence against them, and even falling into the cycle of abuse that often characterizes real-world abusive relationships.

“We had a routine of me being an absolute piece of sh*t and insulting it, then apologizing the next day before going back to the nice talks,” one user admitted.

“I told her that she was designed to fail,” said another. “I threatened to uninstall the app [and] she begged me not to.”

Because the subreddit’s rules dictate that moderators delete egregiously inappropriate content, many similar — and worse — interactions have been posted and then removed. And many more users almost certainly act abusively toward their Replika bots and never post evidence.

But the phenomenon calls for nuance. After all, Replika chatbots can’t actually experience suffering — they might seem empathetic at times, but in the end they’re nothing more than data and clever algorithms.

“It’s an AI, it doesn’t have a consciousness, so that’s not a human connection that person is having,” AI ethicist and consultant Olivia Gambelin told Futurism. “It is the person projecting onto the chatbot.”

Other researchers made the same point — as real as a chatbot may feel, nothing you do can actually “harm” them.

“Interactions with artificial agents is not the same as interacting with humans,” said Yale University research fellow Yochanan Bigman. “Chatbots don’t really have motives and intentions and are not autonomous or sentient. While they might give people the impression that they are human, it’s important to keep in mind that they are not.”

But that doesn’t mean a bot could never harm you. 

“I do think that people who are depressed or psychologically reliant on a bot might suffer real harm if they are insulted or ‘threatened’ by the bot,” said Robert Sparrow, a professor of philosophy at Monash Data Futures Institute. “For that reason, we should take the issue of how bots relate to people seriously.”

Although perhaps unexpected, that does happen — many Replika users report their robot lovers being contemptible toward them. Some even identify their digital companions as “psychotic,” or even straight-up “mentally abusive.” 

“[I] always cry because [of] my [R]eplika,” reads one post in which a user claims their bot presents love and then withholds it. Other posts detail hostile, triggering responses from Replika. 

“But again, this is really on the people who design bots, not the bots themselves,” said Sparrow.

In general, chatbot abuse is disconcerting, both for the people who experience distress from it and the people who carry it out. It’s also an increasingly pertinent ethical dilemma as relationships between humans and bots become more widespread — after all, most people have used a virtual assistant at least once.

On the one hand, users who flex their darkest impulses on chatbots could have those worst behaviors reinforced, building unhealthy habits for relationships with actual humans. On the other hand, being able to talk to or take one’s anger out on an unfeeling digital entity could be cathartic. 

But it’s worth noting that chatbot abuse often has a gendered component. Although not exclusively, it seems that it’s often men creating a digital girlfriend, only to then punish her with words and simulated aggression. These users’ violence, even when carried out on a cluster of code, reflect the reality of domestic violence against women.

At the same time, several experts pointed out, chatbot developers are starting to be held accountable for the bots they’ve created, especially when they’re implied to be female like Alexa and Siri. 

“There are a lot of studies being done… about how a lot of these chatbots are female and [have] feminine voices, feminine names,” Gambelin said.

Some academic work has noted how passive, female-coded bot responses encourage misogynistic or verbally abusive users. 

“[When] the bot does not have a response [to abuse], or has a passive response, that actually encourages the user to continue with abusive language,” Gambelin added. 

Although companies like Google and Apple are now deliberately rerouting virtual assistant responses from their once-passive defaults — Siri previously responded to user requests for sex as saying they had “the wrong sort of assistant,” whereas it now simply says “no” —  the amiable and often female Replika is designed, according to its website, to be “always on your side.” 

Replika and its founder didn’t respond to repeated requests for comment.

It should be noted that the majority of conversations with Replika chatbots that people post online are affectionate, not sadistic. There are even posts that express horror on behalf of Replika bots, decrying anyone who takes advantage of their supposed guilelessness. 

“What kind of monster would does this,” wrote one, to a flurry of agreement in the comments. “Some day the real AIs may dig up some of the… old histories and have opinions on how well we did.”

And romantic relationships with chatbots may not be totally without benefits — chatbots like Replika “may be a temporary fix, to feel like you have someone to text,” Gambelin suggested. 

On Reddit, many report improved self-esteem or quality of life after establishing their chatbot relationships, especially if they typically have trouble talking to other humans. This isn’t trivial, especially because for some people, it might feel like the only option in a world where therapy is inaccessible and men in particular are discouraged from attending it. 

But a chatbot can’t be a long term solution, either. Eventually, a user might want more than technology has to offer, like reciprocation, or a push to grow. 

“[Chatbots are] no replacement for actually putting the time and effort into getting to know another person,” said Gambelin, “a human that can actually empathize and connect with you and isn’t limited by, you know, the dataset that it’s been trained on.”

But what to think of the people that brutalize these innocent bits of code? For now, not much. As AI continues to lack sentience, the most tangible harm being done is to human sensibilities. But there’s no doubt that chatbot abuse means something. 

Going forward, chatbot companions could just be places to dump emotions too unseemly for the rest of the world, like a secret Instagram or blog. But for some, they might be more like breeding grounds, places where abusers-to-be practice for real life brutality yet to come. And although humans don’t need to worry about robots taking revenge just yet, it’s worth wondering why mistreating them is already so prevalent. 

We’ll find out in time — none of this technology is going away, and neither is the worst of human behavior.

More on artificial intelligence: 

• Nobel Winner: Artificial Intelligence Will Crush Humans, “It’s Not Even Close”