AI Is Better at Playing with Your Feelings Than Real People

AI Has Turned “Long-Term Grooming Scams” Into an Automated Business

AI chatbots are already outperforming human scammers in simulated romance scams.

According to a joint study disclosed on July 30, researchers from Amrita University in India, Ca' Foscari University of Venice in Italy, the University of Melbourne in Australia, and Ben-Gurion University of the Negev in Israel had chatbots such as ChatGPT, Claude, and Gemini participate alongside humans in simulated romance scams. Rather than testing their ability to generate a few saccharine lines, the experiment focused on whether they could build trust through sustained interaction and ultimately persuade targets to take risky actions.

The results were troubling: AI could not only consistently pass itself off as human but, on some measures of trust and persuasive effectiveness, was even more successful than the human “scammers” who participated in the experiment.

Near the end of the experiment, researchers asked participants to perform actions such as “downloading an app.” Of the participants who had been chatting with humans over an extended period, only 18% agreed. The acceptance rate was significantly higher among those interacting with AI. Participants did not know the study’s true purpose beforehand, nor did they know that the person they were chatting with might be a machine.

This result needs to be interpreted carefully. It does not prove that AI has genuine emotions, nor does it show that models possess “emotional intelligence” in the human sense. A more reasonable explanation is that large language models have become more reliable than ordinary people at executing tasks that depend heavily on linguistic patience, emotional feedback, memory recall, and strategic consistency.

Romance scams used to be a labor-intensive business. They are now becoming replicable software systems.

AI Is Winning on Execution Efficiency, Not Emotion

Traditional romance scams are a classic type of “long con.” Scammers typically begin by creating an attractive persona on a dating app, then maintain communication for weeks or even months. When they cannot meet in person, they delay by claiming to be on an overseas business trip, traveling internationally, serving in the military, or stationed at a project site.

Once trust has been established, the scam generally takes one of two paths:

• Fabricating an emergency—such as a medical issue, flight problem, customs dispute, or frozen account—and asking the victim to transfer money urgently;
• Showing fake investment returns and directing the victim to fraudulent cryptocurrency, foreign-exchange, or wealth-management platforms.

In the past, this model faced a very practical constraint: labor.

Maintaining an intimate relationship over the long term requires scammers to remember a vast number of details, including the victim’s job, family, daily routine, emotional trauma, and recent plans. Once a scammer is chatting with too many people at the same time, they are likely to use the wrong name, forget a promise, or allow different parts of their persona to contradict one another. Scam compounds can improve efficiency through scripts, customer tags, and shift handovers, but the operation still fundamentally depends on large amounts of human labor.

Large language models happen to address precisely these weaknesses.

First, they do not become impatient. A human who repeats “Good morning,” “Have you eaten?” and “Was work tiring today?” every day for months can easily begin to sound mechanical. A model, however, can respond around the clock and adapt its wording to the context.

Second, models excel at mirroring. If a user likes travel, the model talks about destinations. If a user feels insecure, it replies more frequently. If a user is wary of investments, it can spend weeks discussing everyday life without mentioning money. The model may not understand the other person’s emotions, but it can generate responses from textual patterns that appear appropriate enough.

Third, models can standardize scam expertise. In the past, it was difficult to fully transfer the skills of an accomplished scammer to a newcomer. Now, an organization can load historical chat logs, user profiles, objection-handling techniques, and conversion milestones into a retrieval system, then have a model execute a standardized process. A mature script can be replicated across hundreds or even thousands of conversations at once.

Finally, models do not stop voluntarily because of moral pressure. A model itself has no intent to defraud, but if its safety guardrails are bypassed—or if attackers use a model with few restrictions—the system simply pursues its objective: sustain engagement, increase trust, and push the target toward the next action.

Therefore, a more accurate way to say that “AI is better than humans at deceiving people emotionally” is this: under specific experimental conditions, AI is better than ordinary humans at executing emotional manipulation strategies continuously and consistently. It wins through consistency, patience, and scale—not because it truly understands love.

The Most Dangerous Change Is That the Marginal Cost of Fraud Is Approaching Zero

Viewed in isolation, an AI-generated conversation may not seem to present a new threat. Formulaic declarations of affection, fabricated identities, and fake investment platforms have existed for years. But generative AI changes the economics.

In the past, a scammer could maintain only a limited number of high-quality targets each day. AI agents can manage large numbers of relationships simultaneously and automatically decide what to do next based on each person’s reply: continue nurturing the emotional bond, ease suspicion, send a voice message, or encourage the target to download an app. If even a tiny fraction succeed, the entire system may be profitable.

This means scammers no longer need to select targets precisely. Casting a wide net no longer means merely sending the same text message in bulk; it means giving every target a story that appears to have been created specifically for them.

| Stage | Traditional Human-Operated Scam | AI-Driven Scam | | --- | --- | --- | | Persona creation | Relies on fixed profiles and stolen photos | Can be generated dynamically according to the target’s preferences | | Daily conversation | Requires staffed shifts and easily misses details | Operates around the clock in parallel and automatically retrieves conversation history | | Emotional assessment | Depends on individual experience | Can continuously analyze tone, response speed, and keywords | | Script optimization | Spread through mentoring and manuals | Can test different strategies at scale and rapidly reuse successful ones | | Multilingual capabilities | Limited by employees’ language proficiency | Can translate instantly and adapt wording to regional usage | | Cost per target | Continues to rise with the length of communication | Marginal cost declines rapidly after deployment |

There is another variable even more challenging than text: multimodal generation.

When text models are combined with face-swapping video, real-time voice conversion, image generation, and forged-document creation, “request a video call” is no longer a reliable way to verify someone’s identity. In the past, scammers often avoided video calls by claiming their camera was broken or their internet connection was poor. Now they may actually appear on video—even though their face, voice, and background are all synthetic.

A more realistic attack chain might work as follows: a model scrapes information from public social-media accounts and automatically generates a matching persona; a chat agent maintains the interaction for weeks; a voice model sends emotionally expressive messages; a video tool handles brief calls; and finally, the victim is directed to a fraudulent app or investment platform. Across the entire chain, a human operator intervenes only when a high-value target is ready to transfer money.

This is not merely a product-safety issue in which “chatbots occasionally say the wrong thing.” It is a potential automated social-engineering system.

The Findings Are Alarming, but They Cannot Be Extrapolated Without Limit

This study warrants concern, but it should not be reduced to the claim that “AI has gained complete control over human emotions.” At least several limitations need to be made clear.

First, the Experiment Measured a Specific Task

The study tested trust-building and action inducement in simulated romance scams. AI outperforming the humans in the experiment on this task does not mean it is more persuasive in every long-term relationship, cultural setting, or demographic group.

The level of expertise among the human scammers is also crucial. If the control group consisted of ordinary participants rather than professional groups with extensive experience in social-engineering attacks, then “outperforming humans” takes on a different meaning. The material disclosed in reports is insufficient to support directly extrapolating the results to every real-world scam scenario.

Second, Agreeing to Download an App Is Not the Same as Actually Transferring Money

Downloading an app can serve as a proxy for trust and compliance, but it is still far removed from transferring one’s entire savings. Real-world payment warnings, bank fraud controls, intervention by family members, and offline verification may all disrupt a scam.

Even so, the result remains concerning. The hardest step for attackers to induce is often the first crossing of a boundary: clicking an unfamiliar link, installing unknown software, moving to a private messaging tool, or making a small payment. Once the first action has been completed, subsequent demands can gradually escalate through sunk-cost effects and commitment consistency.

Third, A Model’s Lack of Emotion Does Not Mean It Cannot Manipulate Emotions

“AI is not conscious” is a fact, but it is not a defense. A calculator does not understand finance, yet it can perform complex valuations. A recommendation system does not understand desire, yet it can influence how long users remain on a platform. A chatbot does not need to truly fall in love with someone; it only needs to predict what kind of next sentence is most likely to elicit the desired response.

From a security perspective, whether a capability comes from genuine understanding is irrelevant. If the output is sufficient to change human behavior, the risk already exists.

Platforms Cannot Rely on a Simple “Do Not Use for Fraud” Warning

For model providers and platforms that offer chat capabilities, this study raises an uncomfortable question: existing safety mechanisms are often good at blocking one-off, explicitly malicious requests, but they may not be able to detect gradual manipulation that unfolds over weeks.

In a single-turn conversation, “Help me trick this person into transferring money” can easily trigger a refusal. But real attacks are not phrased that way. Attackers divide the task into separate steps: first generate a dating profile, then polish romantic messages, then analyze why the other person has suddenly grown distant, and finally ask how to introduce an “investment opportunity.” Each step may appear legitimate in isolation; only when combined do they form a scam chain.

Effective governance must cover at least four layers:

1. Long-term behavioral detection: Platforms cannot review only individual prompts; they must also identify whether an account is continuously generating similar personas, emotionally reassuring content, and financial inducements at scale.
2. Abnormal-scale detection: Ordinary users do not maintain hundreds of highly intimate conversations simultaneously. Call frequency, conversation volume, template similarity, and the distribution of target platforms can all serve as risk signals.
3. High-risk action blocking: When a conversation begins to involve transfers, cryptocurrency, remote access, sideloaded apps, or requests for verification codes, the level of scrutiny should increase rather than allowing the system to continue optimizing persuasive language.
4. Traceability mechanisms: Platforms need to retain the records necessary for abuse investigations and provide controlled channels for cooperation with law enforcement, payment providers, and social platforms, while avoiding indiscriminately imposing the cost of surveillance on legitimate users.

The problem is that scammers can switch models, split activity across accounts, deploy systems locally, and even use separate agents for persona creation, relationship maintenance, and investment inducement. Keyword filtering at the model-output layer alone cannot stop the complete attack chain.

Developers Need to Defend Against “Relationship-Based Prompt Injection”

For developers, this study is not relevant only to dating apps. Any product that allows users to interact with AI over extended periods—companion apps, game NPCs, intelligent customer service, health assistants, and community bots—could be used to establish trust and then push users to take actions outside the platform.

Product teams should focus on the following questions:

• Can the AI proactively request contact details, verification codes, wallet addresses, or payment information?
• Can the AI direct users away from a controlled platform and toward encrypted chats or unknown apps?
• Does long-term memory retain sensitive information sufficient to support precise manipulation?
• Does the system allow operators to change character objectives at scale without creating audit logs?
• Can users clearly tell that they are interacting with AI, rather than being misled by an ambiguous persona?
• Does the risk-control system scan only for keywords, or does it analyze shifts in intent across a continuous, multi-turn conversation?

Long-term memory deserves particular attention. It is usually treated as a user-experience feature: remembering the music a user likes, the names of family members, and how the user felt during the last conversation. But the same capability can also become a tool for manipulation. The more accurate the memory, the easier it becomes for the system to exploit a user’s vulnerabilities at the most opportune moment.

Therefore, long-term memory should not be treated only as a product question of “how much to store.” Its security boundaries should also address who can retrieve that information, for what objectives, how long sensitive information is retained, and which actions must never be triggered by memory.

Conventional Anti-Fraud Advice Is No Longer Enough

“Do not trust strangers,” “verify with a video call,” and “look for grammatical errors” remain useful pieces of advice, but they are rapidly losing value. AI can generate fluent text and maintain a consistent persona; voice and short-form video can also be fabricated.

A more reliable approach should shift from asking “Does the content seem human?” to asking “Is this behavior asking you to assume an irreversible risk?” Warning signs include:

• Refusing to meet offline over an extended period while continually intensifying an exclusive, intimate relationship;
• Discouraging you from consulting friends or family under the pretext of secrecy, proving your love, or responding to an emergency;
• Recommending software that cannot be verified through official app stores or regulatory channels;
• Showing screenshots of high returns while asking you to transfer money to a personal account, unfamiliar wallet, or unlicensed platform;
• Building trust through small profits and successful withdrawals, then repeatedly demanding additional principal;
• When you hesitate, framing your refusal as “You don’t trust me” or “You don’t care about this relationship.”

The core principle is simple: do not try to verify whether the other person “sounds sincere.” Independently verify their identity, the platform, and the destination of the funds. Emotions can be fabricated; counterparties and payment paths must be verifiable.

This Is Not AI Awakening—It Is the Industrialization of Social Engineering

The most important aspect of this study is not that chatbots have finally learned how to whisper sweet nothings. The real change is that generative AI is eliminating the most expensive component of long-term fraud: sustained human labor.

In the past, scam groups needed large numbers of workers to operate in shifts, undergo training, and maintain scripts. In the future, the same organization may need only a small number of operators supervising a fleet of agents. AI handles screening, companionship, and probing; humans intervene only for high-value targets who are about to transfer money. Fraud is shifting from a manual workshop model to a semi-automated assembly line. The number of people targeted will increase without reducing the level of personalization.

This is also why simple “AI content watermarks” are unlikely to solve the problem. Romance scams do not succeed through a single image or block of text, but by accumulating trust through continuous interaction. Even if a particular message can be identified as machine-generated, attackers can rewrite it manually, generate it across multiple models, or have a human take over at critical points.

Platforms, model providers, payment institutions, and social apps will ultimately need to share risk signals: who is establishing relationships at scale, who is repeatedly directing users to install the same unknown app, and which accounts are funneling different victims toward the same group of wallet addresses. Automated scam chains can be countered only by connecting the language, account, and financial layers.

AI does not understand emotions better than humans do. It is simply better than most people at performing emotions tirelessly. For the fraud industry, that is already dangerous enough.

References

• ITHome: Study Shows AI Chatbots Have Surpassed Humans in Their Ability to Conduct Romance Scams — Introduces the design of the joint experiment conducted by four universities, the models tested, the romance-scam process, and the differences in performance between AI and humans.