When ChatGPT and Midjourney both made their public debuts in 2022, it was unclear just how rapid and far-reaching their adoption would be. Naysayers claimed that their use cases would remain niche, while proponents argued that virtually every industry would eventually see some kind of effect from the technology.
It’s since become clear that the latter camp was right: For better or worse, 2023 will go down in history as the year generative AI officially entered the global zeitgeist. And it didn’t take long for bad actors to begin leveraging these tools in a plethora of different ways to commit fraud.
Today, almost every business needs a game plan for how it will deal with AI-generated text, photos, videos, and audio. This is especially true for online marketplaces and other online platforms for whom trust and safety is a major concern.
Left unchecked, fake profiles, fake product listings, and other fake content can all cause serious hardship for your users and irreparably damage the hard-fought trust that you have built. At best, generative AI fraud may cause your users to think twice before completing a transaction on your platform; at worst, it may send them into the arms of your competitors.
Below, we take a closer look at the types of businesses that are at risk, as well as the different ways that fraudsters are adding AI to their toolkits. We also offer guidance that you can use to make your business more resilient to the threats of generative AI.
Who’s at risk?
Any online marketplace or platform can experience generative AI fraud. But there are certain risk factors that make a business potentially more vulnerable.
More automated verifications. Automating your buyer/seller identity verification processes can lead to faster onboarding with fewer human errors. But it isn’t foolproof, and vulnerabilities can still exist. Bad actors who identify a weak point in a platform’s automated verification process can exploit it to open a large number of accounts quickly, become further ingrained in your platform, and potentially carry out a higher volume of fraud.
Frequent and/or large payouts. If your online marketplace deals in the sale of high-value items or you pay out to sellers frequently, you may find yourself at a greater risk of being targeted by bad actors. After all, a fraudster can make more money defrauding a customer on a high ticket than a low ticket, and more frequent payouts offer the fraudster more opportunities to take the money and run before being caught.
Presence of “power users.” Power users — sellers who lean heavily into your marketplace, attract business, and handle a high volume of transactions — are an asset to your business. But they can also pose a risk. If those power users were to begin engaging in fraud, or if their accounts were compromised by bad actors, their reach might empower them to inflict a lot of damage in a short period of time before they are caught and shut down.
Scale. The more buyers, sellers, and product listings your marketplace has, the more content there is for moderators to sift through and evaluate — and the easier it is for a fraudster to blend into the background and avoid detection until they’ve successfully defrauded legitimate users.
How are criminals using generative AI to commit fraud?
With generative AI, it’s never been easier for fraudsters to create a wide range of assets they can leverage to engage in marketplace fraud. Some commonly seen examples include:
Phishing emails
Large language models (LLMs) can be used to create phishing emails and other messages fraudsters can use to compromise accounts on your marketplace. Once compromised, those accounts can be used to engage in fraud.
What’s more, fraudsters are no longer limited by language or time: In an instant, it’s possible to create hundreds or thousands of grammatically-correct phishing emails in dozens of languages, making them harder to spot and avoid. According to a recent study, the first two months of 2023 saw a 135% increase in never-before-seen phishing email attempts — corresponding with the release and rapid adoption of popular, publicly available LLMs.
AI-generated selfies
Generative AI isn’t just limited to text. Models like Midjourney, DALL-E, and others can now be used to create hyper-realistic images that even the trained eye may find difficult to detect. AI-generated selfies are particularly dangerous, as they might be leveraged in:
- Fake IDs and documents
- Selfie verification
- Profile photos on seller accounts
- and more
Deepfakes
Deepfakes are fake photos or videos of real people that can be created by feeding sample images into deepfake software. With deepfakes, a bad actor can take real photos from real people — for example, by scraping them from a social media site — and use those photos to generate brand-new images or videos of that person.
These assets can then be used to try and skirt around identity verification efforts — for example, to try and bypass selfie verification, video verification, and even live video interviews during customer onboarding.
AI-generated audio
AI-generated audio works the same way as deepfakes. A bad actor feeds a sample clip of somebody’s voice into software, provides the software with a script — often generated using a tool like an LLM — and it generates new audio of that person saying something they never actually said. This audio can then be leveraged in phishing attacks, blackmailing attempts, and more.
And it doesn’t take much input to accurately recreate someone's voice. Microsoft’s VALL-E model, for example, can simulate a voice with as little as three seconds of input. With video so prevalent on social media, it’s relatively easy for a bad actor to find the assets they need to generate new audio.
Synthetic IDs
Synthetic IDs are fake identities created by combining real and fake information. A real name, Social Security number, and birthday can be paired with a fake address and contact information, for example, to create a synthetic ID that is harder to detect than a purely fabricated one. Once the basis of a synthetic ID is established, generative AI can be used to fill in the gaps — creating IDs, documents, profiles, and other assets.
How to prevent AI fraud
As you can see, bad actors are capable of using generative AI in a plethora of different ways to commit marketplace and other types of fraud. Due to this variety, there is no single silver bullet that solves for every use case.
Instead of looking for that Holy Grail, it often makes sense to build overlapping layers of redundancy into your identity verification processes and anti-fraud strategy.
Ensuring secure data capture
As AI-generated selfies, audio, and video have become more and more realistic in recent years, much of the focus in the anti-fraud space has focused on answering one question: How do you discern between legitimate and AI-generated assets?
While that’s an important question to ask, we don’t think it’s the most important. Instead, we would ask: How do you prevent fraudsters from submitting AI-generated assets during a verification flow to begin with? After all, if a bad actor can’t upload their fake photos, videos, and audio, it doesn’t matter how convincing they are.
Solving the problem of secure data capture can take many forms. Some solutions include:
- Auto-capturing photos, video, and audio throughout a session, which can then be compared against the assets uploaded by the user or analyzed for fraud signals
- Detecting the use of a virtual camera which may have allowed for an AI-generated image to be uploaded
- Detecting other forms of user tampering that may compromise a session
Collecting passive signals to assess session risk
When a user uploads a photo, answers a question, or provides some other kind of information during the verification process, those are known as active signals because the user is actively providing them to you. But while they’re important, they’re not the only type of data you can collect and use to assess a session’s risk.
Passive and behavioral signals — which are collected automatically in the background of a session, often without a user’s awareness — can also be a treasure trove of risk signals that give you a more accurate idea of the risk posed by a given user. And because they’re collected in the background, instead of being provided directly by the user, they’re a lot harder (though not impossible) to fake.
There are many passive and behavioral signals you might want to collect and factor into your risk assessment calculations. Some of the most important include:
- IP address
- Device fingerprint
- Browser fingerprint
- Use of a VPN
- Use of developer tools
- Hesitation
- Distraction events
- Mouse clicks
- Keyboard strokes
Leveraging multiple forms of verification for visual analysis
When it comes to verifying a user’s identity using visual assets — for example, a government ID, selfie, or video — the more the merrier. Each input you collect provides you with more visual data that you can use to make comparisons and draw conclusions about your users’ risk profile.
It also gives the bad actor extra hurdles to jump through, which can dramatically lower their chances of success.
It’s relatively easy, for example, for a bad actor to create a fake ID using an AI-generated selfie. It’s much harder to create that fake ID and then have to create a second image that is capable of passing a liveness check for use during selfie verification.
Of course, requiring a user to complete multiple forms of verification does add friction to the sign-up process, and could in some instances be costly. With progressive risk segmentation, you can tailor the verification flow to each user depending on how much risk you detect in real time. For low-risk users, you might only require government ID verification, for example. But for high-risk users, you might pair that with selfie or database verification.
Leveraging outside data for identity binding
Finally, it’s important to note that identity verification doesn’t need to be completed using only data that is actively or passively provided by your user. Third-party data from authoritative data sources such as an issuing database can go far in increasing your assurance that a person is who they say they are.
For example, if you collect a driver’s license during onboarding, you might also choose to query the AAMVA database to ensure the information contained in that license matches official records. While this will not help you to directly detect AI-generated content, it can still be very effective in identifying fraudulent IDs.
Fight generative AI fraud with Persona
The long-term societal effects of generative AI tools will take years to understand. But one thing is clear: Bad actors are already using them to carry out a variety of fraud. AI is now most likely a permanent fixture in fraudsters’ toolkits — and something that online marketplaces need to account for.
Combating this threat requires a robust and comprehensive anti-fraud strategy — one that relies on a variety of risk signals, verification methods, and built-in redundancy that makes it more difficult for a bad actor to skate through even with the help of AI-generated assets.
Interested in learning more? Try Persona for free or get a demo today.
