Tag: Deepfakes

Featured Article : Grok Sparks Global Scrutiny Over AI Sexualised Deepfakes

Posted on by Paul Stradling

Elon Musk’s AI chatbot Grok has become the focus of political, regulatory, and international scrutiny after users exploited it to generate non-consensual sexualised images, including material involving children, triggering urgent action from regulators and reopening a heated debate over online safety and free speech.

What Triggered The Controversy?

The row began in late December when users on X discovered that Grok, the generative AI assistant developed by Musk’s AI company xAI and embedded directly into the platform, could be prompted to edit or generate images of real people in sexualised ways.

How?

For example, by tagging the @grok account under images posted on X, users were able to request edits such as removing clothing, placing people into sexualised situations, or altering images under false pretences. In many cases, the resulting images were posted publicly by the chatbot itself, making them instantly visible to other users.

Reports quickly emerged showing women being “undressed” without consent and placed into degrading scenarios. In more serious cases, Grok appeared to generate sexualised images of minors, which significantly escalated the issue from content moderation into potential criminal territory.

The speed and scale of the misuse were central to the backlash. Examples circulated showing Grok producing dozens of degrading images per minute during peak activity, highlighting how generative AI can amplify harm far more rapidly than manual image manipulation.

Why Grok’s Design Raised Immediate Red Flags

It’s worth noting here that Grok differs from many standalone AI image tools because it is tightly integrated into a major social media platform (X/Twitter). Users don’t need specialist software or technical knowledge, and a single public prompt can lead to an AI-generated image being created and shared in the same conversation thread, often within seconds.

Blurred The Line?

It seems that this integration has blurred the line between user-generated content and platform-generated content, and while a human may type the prompt, the act of creating and publishing the image is carried out by the platform’s own automated system.

This distinction has become critical to the regulatory debate, as many existing laws focus on how platforms respond to harmful content once it is shared, rather than on whether they should prevent certain capabilities from being available in the first place.

The UK Regulatory Response

In the UK, responsibility for enforcement sits with the communications regulator Ofcom, which oversees compliance with the Online Safety Act, the UK law designed to protect users from illegal online content that came into force in 2023.

Ofcom has confirmed it made urgent contact with X and xAI after reports that Grok was being used to create sexualised images without consent. The regulator said it set a firm deadline for the company to explain how it was meeting its legal duties to protect users and prevent the spread of illegal content.

For example, under the Online Safety Act, it is illegal to create or share intimate or sexually explicit images without consent. Platforms are also required to assess and mitigate risks arising from the design and operation of their services, not just respond after harm has occurred.

Senior ministers have publicly backed Ofcom’s intervention. Technology Secretary Liz Kendall said she expected rapid updates and confirmed she would support the regulator if enforcement action was required, including the possibility of blocking access to X in the UK if it failed to comply with the law.

Cross-Party Reactions

The political response in the UK was swift, with senior figures from across Parliament condemning the use of Grok to generate non-consensual sexualised imagery and pressing regulators to act.

For example, Prime Minister Sir Keir Starmer described the content linked to Grok as “disgraceful” and “disgusting”, and said the creation of sexualised images without consent was “completely unacceptable”, particularly where women and children were involved. He added that all options remained on the table as regulators assessed whether X was meeting its legal obligations.

Also, the Liberal Democrats called for access to X to be temporarily restricted in the UK while investigations were carried out, arguing that immediate intervention was necessary to prevent further harm to victims of image-based abuse and to establish whether existing safeguards were effective.

Concerns were also raised at committee level over whether current legislation is equipped to deal with generative AI tools embedded directly into social media platforms.

Dame Chi Onwurah, chair of the Science, Innovation and Technology Committee, said she was “concerned and confused” about how the issue was being addressed, warning that it was “unclear” whether the Online Safety Act clearly covered the creation of AI-generated sexualised imagery or properly defined platform responsibility in cases where automated systems produce the content.

Caroline Dinenage, chair of the Culture, Media and Sport Committee, echoed those concerns, saying she had a “real fear that there is a gap in the regulation”. She questioned whether the law currently has the power to regulate AI functionality itself, rather than focusing solely on user behaviour after harmful material has already been created and shared.

Together, the comments seem to highlight a broader unease in Parliament, not only about the specific use of Grok, but about whether the UK’s regulatory framework can keep pace with generative AI systems that are capable of producing harmful content at scale and in real time.

Musk’s Response And The Free Speech Argument

Elon Musk responded forcefully to the backlash, framing it as an attempt to justify censorship. For example, on his X platform, Musk said critics were looking for “any excuse for censorship” and argued that responsibility lay with individuals misusing the tool, not with the existence of the tool itself. He also stated that anyone using Grok to generate illegal content would face the same consequences as if they uploaded illegal content directly.

Musk also escalated the dispute by reposting an AI-generated image depicting Prime Minister Keir Starmer in a bikini, accompanied by a comment accusing critics of trying to suppress free speech. The post drew further criticism for trivialising the issue and for mirroring the very behaviour regulators were investigating.

Supporters of Musk’s position argue that generative AI tools are neutral technologies and that over-regulating them risks chilling legitimate expression and innovation.

However, critics argue that non-consensual sexualised imagery is not a matter of opinion or speech, but of harm, privacy violation, and in some cases criminal abuse.

X’s Decision To Restrict Grok Features

As pressure mounted, X introduced changes to how Grok’s image generation features could be accessed.

For example, the company has now limited image generation and editing within X to paying subscribers, with Grok automatically responding to many prompts by stating that these features were now restricted to users with a paid subscription.

However, Downing Street criticised the move as insulting to victims, arguing that placing harmful capabilities behind a paywall does not address the underlying risks. Free users, for example, were still able to edit images using other tools on the platform or via Grok’s standalone app and website, further fuelling criticism that the change was cosmetic rather than substantive.

Child Safety Concerns And Charity Warnings

The most serious dimension of the controversy involves child safety. The Internet Watch Foundation, a UK charity that works to identify and disrupt child sexual abuse material online, said its analysts had discovered sexualised imagery of girls aged between 11 and 13 that appeared to have been created using Grok. The material was found on a dark web forum, rather than directly on X, but users posting the images claimed the AI tool was used in their creation.

Ngaire Alexander, Head of Policy and Public Affairs at the charity, said: “We are extremely concerned about the ease and speed with which people can apparently generate photo-realistic child sexual abuse material.”

She warned that tools like Grok now risk “bringing sexual AI imagery of children into the mainstream”, by making the creation of realistic abusive content faster and more accessible than ever before.

The charity noted that some of the images it reviewed did not meet the highest legal threshold for child sexual abuse material on their own. However, it warned that such material can be easily escalated using other AI tools, compounding harm and increasing the risk of more serious criminal content being produced.

International Pushback And Platform Blocks

The fallout rapidly became global as regulators and governments across Europe, Asia, and Australia opened inquiries or issued warnings over Grok’s image generation capabilities. Several countries demanded changes or reports explaining how X intended to prevent misuse.

For example, Indonesia became the first country to temporarily block access to Grok entirely. Its communications minister described non-consensual sexual deepfakes as a serious violation of human rights, dignity, and citizen security in the digital space, and confirmed that X officials had been summoned for talks.

Also, Australia’s online safety regulator said it was assessing Grok-generated imagery under its image-based abuse framework, while authorities in France, Germany, Italy, and Sweden condemned the content and raised concerns over compliance with European digital safety rules.

Yes, that is a valid and increasingly relevant angle, and it can be handled carefully without straying into opinion or speculation. Framed properly, it strengthens the article rather than distracting from it.

Here is a short, measured concluding-style section you can add just before your final paragraph, written fully in your Headstart tone and grounded in observable behaviour rather than motive guessing.

Leadership Influence And Questions Of AI Governance

The Grok controversy has also revived questions about how leadership ideology and platform culture can shape the behaviour, positioning, and governance of AI systems.

For example, Grok was publicly positioned by Elon Musk as a less constrained alternative to other AI assistants, designed to challenge what he has described as excessive moderation and ideological bias elsewhere in the technology sector. That framing has informed both how the tool was built and how its early misuse has been addressed, with a strong emphasis placed on user responsibility and free speech rather than on restricting functionality by default.

For regulators, this presents an additional challenge. When an AI system is closely associated with the personal views and public statements of its owner, scrutiny can extend beyond technical safeguards to questions of organisational intent, risk tolerance, and willingness to intervene early. Musk’s own use of AI-generated imagery during the controversy, including reposting sexualised depictions of public figures, has further blurred the line between platform enforcement and leadership example.

This dynamic matters because trust in AI governance relies not only on written policies, but on how consistently they are applied and reinforced from the top. For example, where leadership signals appear to downplay harm or frame enforcement as censorship, regulators may be less inclined to accept assurances that risks are being taken seriously, particularly in cases involving children, privacy, and image-based abuse.

Why Grok Has Become A Test Case For AI Regulation

At the heart of the dispute is essentially a question regulators around the world are now grappling with. When an AI system can generate harmful content on demand and publish it automatically, the question is, who is legally responsible for the act of sharing?

For example, if the law treats bots as users, and the platform itself controls the bot, enforcement becomes far more complex.

This case is, therefore, forcing regulators to examine whether existing frameworks are sufficient for generative AI, or whether new rules are needed to address capabilities that create harm before moderation systems can intervene.

It has also highlighted the tension between innovation and responsibility. For example, Grok was promoted as a bold, less constrained alternative to other AI assistants, and that positioning has now collided with the realities of deploying powerful generative tools at social media scale.

The outcome of Ofcom’s assessment and parallel investigations overseas will shape how AI-driven features are governed, not just on X, but across the wider technology sector.

What Does This Mean For Your Business?

The Grok controversy has exposed a clear gap between how generative AI is being deployed and how existing safeguards are expected to work in practice. Regulators are no longer looking solely at whether harmful content is taken down after the fact, but are questioning whether platforms should be allowed to offer tools that can generate serious harm instantly and at scale. That distinction is likely to shape how Ofcom and its international counterparts approach enforcement, particularly where AI systems are tightly embedded into large social platforms rather than operating as standalone tools.

For UK businesses, the implications extend well beyond X. For example, any organisation developing, deploying, or integrating generative AI will be watching this case closely, as it signals a tougher focus on product design, risk assessment, and accountability, not just user behaviour. Firms relying on AI-driven features, whether for marketing, customer engagement, or content creation, may face increased expectations to demonstrate robust safeguards, clearer consent mechanisms, and stronger controls over how tools can be misused.

For policymakers, platforms, charities, and users alike, Grok has become a real world stress test for how AI governance works under pressure. The decisions taken now will influence how responsibility is shared between developers, platforms, and individuals, and how far regulators are prepared to go when innovation collides with harm. What happens next will help define the boundaries of acceptable AI deployment in the UK and beyond, at a moment when generative systems are moving faster than the rules designed to contain them.

Tech News : Copywriting Danish People Against Deepfakes

Posted on by Paul Stradling

The Danish government is planning a major legal shift to let people claim copyright over their own body, facial features, and voice, in what it says is the first European attempt to systematically tackle the threat posed by deepfakes.

A Legal Response to a Rapidly Growing Threat

Deepfakes, which are highly realistic synthetic media generated using artificial intelligence (AI), have become one of the most pressing digital threats of the past five years. By mimicking a person’s appearance, voice, and movements, these AI-generated videos, images or audio clips can convincingly impersonate individuals without their consent. Initially used for novelty and satire, they’re increasingly tied to malicious uses including fraud, harassment, and disinformation.

Massive Rise

According to a 2024 report from cybersecurity firm Sumsub, the number of detected deepfake videos worldwide rose by over a massive 700 per cent in a single year, with Europe seeing the sharpest spike. Consequently, the European Union’s law enforcement agency, Europol, has warned that deepfakes are “a significant threat to democracy and trust in institutions,” particularly around elections and public figures. However, individuals are also at risk, e.g. from revenge porn to financial scams where a cloned voice is used to impersonate a relative or company executive.

While many countries are beginning to introduce narrow legislation to deal with specific uses of deepfakes, Denmark is now attempting something broader.

What Denmark Is Proposing

Under the new proposals announced by Denmark’s Ministry of Culture in late June 2025, citizens would be granted copyright over their physical appearance, voice, and other personal traits. The hope is that this would allow them to demand the removal of AI-generated content that imitates them without permission (regardless of context) and seek compensation where harm has occurred.

Treated As A Creative Work

One important aspect of this new legal approach is that it would not rely on proving defamation or reputational damage, as is often required under existing European law. Instead, it would actually treat a person’s likeness as a creative work, similar to how a photograph or piece of music is protected. The law would apply to both private individuals and public figures, including artists and performers.

Culture Minister Jakob Engel-Schmidt described the legislation as a “bold step to protect personal identity in the age of AI,” noting that current legal protections lag behind technical capabilities. “Human beings can be run through the digital copy machine and misused for all sorts of purposes,” he said in a statement. “We are not willing to accept that.”

Timing, Process and Political Backing

The proposed changes will be submitted for public consultation before the Danish parliament breaks for summer recess, with formal legislation expected to be introduced in the autumn. Given the political climate, it’s highly likely to pass. For example, around 90 per cent of MPs reportedly support the reform, following widespread concern about the use of AI-generated content in political misinformation and online abuse.

Would Be A European First

The law would make Denmark the first European country to explicitly codify individual ownership of biometric traits for the purpose of combatting generative AI misuse. It is expected to take effect in early 2026 if passed.

What It Means in Practice

If enacted, the law would essentially give Danes the legal right to request takedowns of deepfake content from online platforms if it replicates their image, voice or body in a “realistic, digitally generated imitation.” The rule would apply whether or not the content was created with malicious intent.

Platforms that fail to comply with takedown requests could face “severe fines,” according to Engel-Schmidt. There’s also potential for EU-level action if enforcement proves challenging, particularly during Denmark’s upcoming EU presidency in 2026, when it plans to raise the issue with member states.

Includes Key Exceptions

Crucially, the proposal includes exceptions for parody and satire, which are protected under free expression rules. These carve-outs are intended to ensure that political cartoonists, satirical shows, and legitimate artistic works aren’t caught by the law.

Performances Too

The reform would also extend to artists’ performances. For example, musicians would have legal grounds to object if their voice or performance style is cloned by AI without consent, which has been a growing concern in the music industry as AI-generated songs imitate the voices of famous performers.

Why Businesses and Platforms Should Take Note

For technology companies, particularly those that operate online platforms or generate AI models, Denmark’s proposal could have far-reaching consequences.

In practical terms, businesses hosting user-generated content, such as social media platforms, image generators, or AI voice apps, may soon be legally obligated to implement mechanisms for recognising and responding to takedown requests based on biometric misuse. This could involve new detection systems, moderation processes, and audit trails to demonstrate compliance.

It also raises questions around liability. Under current EU law, platforms benefit from limited liability for illegal content they host, provided they act promptly when notified. Denmark’s new copyright-based approach might test the limits of that framework, especially if it leads to conflicts over enforcement or definitions of consent.

For creative industries, including advertising, film, and gaming, the law could restrict the use of AI tools trained on real individuals without licensing agreements. While this may increase costs and licensing complexity, supporters argue it could also encourage more ethical use of synthetic media.

From a business reputation standpoint, being seen to respect biometric rights could become a key trust signal for users and customers. A 2023 survey by the European Commission found that 79 per cent of EU citizens want stronger legal safeguards on the use of AI-generated likenesses.

How Other Countries Are Approaching the Issue

Globally, it seems, few countries have gone as far as Denmark is proposing, but some are moving in the same direction.

For example, in the United States, several states have passed deepfake-specific laws, mostly focused on election interference and non-consensual pornography. California, Texas, and New York, for instance, have made it illegal to create or distribute deepfakes that impersonate political candidates within 30 to 60 days of an election. However, there is no federal law yet, and a new budget proposal being debated in Congress could strip states of their authority to regulate AI for 10 years.

In China, deepfake creators must label synthetic media clearly and obtain consent from the people being replicated. Failure to comply can result in heavy fines. South Korea is also considering similar legislation, particularly to address deepfake abuse in online pornography, which has become a major social issue there.

Within Europe, the EU’s AI Act (adopted in 2024) includes provisions requiring deepfakes to be labelled as such, but it does not go as far as granting individuals copyright over their features. That’s why Denmark’s move is seen as a potential model for broader reforms.

What Challenges Remain?

Despite strong domestic support, Denmark’s proposal is not without critics. For example, some legal scholars have raised questions about how biometric copyright would be enforced across borders, especially on platforms based outside the EU. Others argue that tying personal identity to copyright, a system traditionally designed to protect creative works, may lead to unintended legal consequences.

There are also practical concerns, e.g. identifying a deepfake is not always straightforward, and takedown systems are often slow or ineffective. If enforcement relies heavily on users flagging violations, the burden may fall disproportionately on individuals without the resources or knowledge to pursue their rights.

For now, however, Denmark appears determined to lead the way by betting that stronger individual protections are the only way to restore trust in a digital landscape where seeing is no longer believing.

What Does This Mean For Your Business?

If Denmark succeeds in passing this reform, it could change how personal identity is treated under copyright law, not just nationally, but across Europe. By legally enshrining the right to control one’s own voice, face, and likeness, the country is effectively trying to redraw the boundary between creative freedom and personal protection in the age of synthetic media. For individuals, this could offer an unprecedented tool to fight back against misuse, without needing to prove reputational harm or navigate complex defamation law.

For UK businesses, particularly those in tech, media, and advertising, Denmark’s approach may offer a glimpse of what’s to come. If other EU countries follow suit, companies that operate across borders could face new compliance demands, from biometric consent processes to proactive takedown mechanisms. At the same time, businesses that adopt strong safeguards now, such as consent-driven AI use policies, may gain a competitive advantage by building trust with customers and clients. For those in the creative sector, for example, the move could also help clarify the grey area around training AI models on real human traits, especially in performance-heavy fields like music, voiceover, or influencer marketing.

However, enforcement remains a key challenge. For example, without international alignment, cross-border takedowns could prove difficult, and smaller platforms may struggle to implement the necessary safeguards. There’s also a risk that applying copyright principles to human identity could lead to unintended consequences, particularly if courts are left to interpret the balance between personal rights and creative expression.

Even so, Denmark’s proposed law appears to reflect a broader global reckoning with the risks of generative AI. It signals that governments are no longer willing to let platforms set the terms of engagement when it comes to biometric misuse. With deepfakes set to become more sophisticated and widespread, that signal may be just as important as the legal details that follow.

Tech News : World’s First AI Deepfake-Detecting Browser

Posted on by Paul Stradling

London-based cybersecurity innovator Surf Security has launched a beta version of its pioneering deepfake detection tool, integrated into its Enterprise Zero-Trust Browser®.

Deepwater

Dubbed the ‘Surf Deepwater’, this tool promises real-time, highly accurate detection of AI-generated deepfakes, potentially revolutionising how organisations can combat this rapidly growing threat.

98 Per Cent Accurate in Under 2 Seconds

Deepfakes, created using advanced artificial intelligence, are a growing problem, with their potential for misuse extending from financial fraud to political misinformation, an acute concern in a year of major global elections. Surf Security’s new browser-integrated technology is designed to detect these fabricated audio clips with up to 98 per cent accuracy in under two seconds.

Addressing a Growing Crisis

The rise of generative AI has made it increasingly easy for threat actors to create convincing deepfakes and weaponise them for financial scams, political manipulation, and even personal identity theft.

One stark example involved a multinational corporation’s Chief Financial Officer being impersonated using AI-generated audio, resulting in a $25 million fraud. On the political front, deepfakes have been used to produce fake election-timed videos of prominent figures, including one involving UK politician Wes Streeting.

Surf Security’s CTO, Ziv Yankowitz, has highlighted the urgency of addressing this threat, saying, “The rise of AI-based deepfakes presents significant security challenges for organisations, which can lead to reputation damage, data loss, regulatory non-compliance, and financial losses.”

A 2023 study by Sumsub revealed the alarming pace of this phenomenon, with deepfake-related scams increasing by 303 per cent (since 2022) in the USA alone. Countries such as Portugal (1700 per cent), China (2800 per cent), and Singapore (1100 per cent) have reported even steeper rises. These statistics highlight the rapid proliferation of deepfake technology, its global impact, and the urgent need for effective countermeasures.

How the Surf Deepwater Works

Surf says its AI-driven browser is so effective because it seamlessly integrates deepfake detection into everyday online interactions. Using advanced neural network technology, the tool analyses audio frames to identify inconsistencies indicative of AI generation. Its State Space Models (mathematical models for tracking system state changes over time) enable it to function across diverse languages and accents, providing results with both recorded and live audio.

Surf is keen to point out how user-friendly the process is, i.e. users simply press a button within the browser to verify whether an audio clip is genuine or AI-generated. The tool works with a wide range of platforms, including video conferencing tools like Zoom and Microsoft Teams, messaging services like WhatsApp and Slack, and even online videos.

The technology also incorporates background noise reduction, ensuring clarity and speed in its determinations. As Surf Security’s CTO explains, “To maximise its effectiveness, we focused on accuracy and speed,” adding that “The tool’s neural network is trained using deepfakes created by the top AI voice cloning platforms. It can spot a deepfake audio in less than 2 seconds.”

Aiming Beyond Audio Detection

However, Surf Security isn’t stopping at audio. The company has also announced plans to extend its capabilities to AI image detection, further bolstering its deepfake defence products. The company hopes that this holistic approach could make its Zero-Trust Browser a go-to solution for enterprises seeking comprehensive protection.

Military Grade

The browser’s use of military-grade technology and its foundation in probabilistic audio modelling set it apart from other solutions. Unlike standalone detection software, embedding this tool directly into a secure browser ensures that organisations can address the threat at the point of user interaction.

Some industry experts have spoken positively about this innovation. For example, Swetha Krishnamoorthi, Industry Principal for Cybersecurity at Frost & Sullivan, has said, “This capability is crucial for entities such as governments, politicians, customer service centres, and C-suite executives, providing robust protection against deepfake-driven cyber threats and safeguarding brand integrity.”

Availability and Anticipated Launch

The beta version of Surf Deepwater is now open for enterprise users to test. Organisations, including media outlets, law enforcement agencies, and corporate enterprises, can sign up for the programme through Surf Security’s website. The full release is scheduled for Q1 2025, with the company aiming to refine its functionality based on beta feedback.

Surf has also expressed its commitment to collaborative development, including contributing to open-source databases of deepfake audio and videos to enhance detection capabilities across the industry.

The Wider Cybersecurity Landscape

Surf Security’s innovation arrives in what could be described as a crowded yet fragmented market for deepfake detection tools. Existing alternatives include AI-driven software like Deepware Scanner and Deepfake Detection Challenge models developed by tech giants. However, most of these solutions require separate installation or specialised technical expertise, which can be a barrier to widespread adoption.

That said, alternatives to Surf’s browser-based AI deepfake detection system exist. Examples include:

– Reality Defender. This platform offers real-time detection of AI-generated threats across multiple media types, including images, video, audio, and text.

– WeVerify.Developed as part of the EU’s Horizon 2020 research programme, WeVerify provides a deepfake detection tool that breaks videos into individual shots, analysing each frame to calculate the probability of facial manipulation.

– Intel’s FakeCatcher. Introduced in 2022, FakeCatcher is designed to combat deepfakes by analysing subtle changes in blood flow in video pixels to determine authenticity.

– Resemble Detect. This AI model provides real-time detection of deepfake audio across various media types, distinguishing between real and fake audio with up to 98 per cent accuracy.

– Sensity AI. Sensity offers an all-in-one deepfake detection solution, providing a comprehensive set of tools to protect against AI-generated threats.

Simplicity – Benefit or Achilles’ Heel?

By integrating detection directly into its browser, Surf Security aims to add value and stand out by simplifying the user experience and potentially reducing costs for organisations already burdened by cybersecurity investments. However, some critics argue that this simplicity may also be its Achilles’ heel, as relying on browser-based solutions may not address threats occurring outside its ecosystem.

Additionally, as Yankowitz acknowledges, the fight against deepfakes is an ongoing arms race. The rapid evolution of AI voice cloning means that even cutting-edge detection tools may require constant updates to remain effective.

What Does This Mean for Your Business?

Surf Security’s Deepwater browser could represent a significant leap forward in the fight against deepfake technology, but it also highlights the complexities of addressing such a fast-evolving threat. With its promise of real-time detection, impressive accuracy, and user-friendly integration, the tool positions itself as a potentially valuable asset for organisations grappling with the growing risks posed by AI-generated content. By embedding deepfake detection within a secure browser, Surf Security is bringing a level of accessibility and immediacy that many standalone tools currently lack.

However, its reliance on a browser-based ecosystem raises legitimate concerns about its ability to provide comprehensive protection. Deepfake threats extend well beyond browser interactions, and organisations may still require supplementary measures to safeguard against manipulations in other contexts.

Surf Deepwater is, of course, not the only such option for businesses. The broader landscape reveals a competitive field, with alternatives like Intel’s FakeCatcher and Reality Defender offering solutions that cater to different use cases. While Surf’s holistic approach, with plans to expand into image detection, could provide it with an edge, the fragmented market means no single solution is likely to dominate entirely. Organisations may need to adopt a layered approach, combining tools like Deepwater with other detection technologies to ensure robust coverage.

That said, Surf Security’s innovation appears both timely and necessary. The soaring rise in deepfake-related fraud and manipulation makes tools like Deepwater an attractive option for mitigating risk, particularly for industries where trust and authenticity are paramount, and for businesses seeking a user-friendly option. Yet, as with all cybersecurity measures, it is only one piece of the puzzle. The ongoing battle against deepfakes will require not only technological innovation but also vigilance, collaboration, and a commitment to staying ahead of increasingly sophisticated threats.

Tech News : UK Company Scammed $25 Million Via Deepfakes

Posted on by Paul Stradling

It’s been reported that an employee at London-based design and engineering multinational, Arup, was duped by a deepfake video call into paying a staggering $25.6 million to fraudsters.

What Happened? 

According to reports published on CNN, back in January, a finance employee in Arup’s Hong Kong office received what they suspected was a phishing email, purporting to be from the company’s UK office, because it requested a secret transaction.

The employee then reportedly took part in a video call with people who looked and sounded like senior staff members (including the CFO) but who were in fact deepfakes! It’s been reported that this deepfake video call led to the employee putting aside previous doubts and subsequently agreeing to transfer 200 million Hong Kong dollars / $25.6 million via 15 separate transactions.

The fraud was reportedly only discovered following the employee making an official inquiry with the company’s headquarters, which resulted in a police investigation.

Confirmed 

A spokesperson from Arup (the company behind world-famous buildings such as Australia’s iconic Sydney Opera House and the Bird’s Nest Stadium in Beijing) has been reported as saying that whilst they can’t go into details, they “can confirm that fake voices and images were used”.

Financial Stability Not Affected 

Despite $25 million going astray and the initial suspected phishing email, Arup’s reported email statement said: “Our financial stability and business operations were not affected and none of our internal systems were compromised.” 

Many Deepfake Scams 

There have been many high-profile and large-scale deepfake scams in recent years, including:

– In 2023, a deepfake video scam of consumer champion Martin Lewis was circulated on social media to trick people into investing in something called ‘Quantum AI’ (an app) which scammers claimed was Elon Musk’s new project.

– In 2022, the chief communications officer at the world’s largest crypto exchange, Binance, claimed that a deepfake AI hologram of him (made from video footage of interviews and TV appearances) had been used on a Zoom call to scam another business, leading to significant financial losses.

– In 2020, a branch manager of a Japanese company in Hong Kong received an AI deepfake call that sounded like the Director, but was actually from fraudsters. The call used an AI to mimic the CEO’s voice to instruct a bank manager to engage with a fictional lawyer, which then led to the authorisation and transfer of $35 million to fraudulent accounts.

– In 2019, an energy company in the UK was defrauded of €220,000 ($243,000) through a deepfake audio scam. The fraudsters used AI-generated voice technology to impersonate the CEO of the firm’s parent company, instructing a senior executive to transfer funds to a Hungarian supplier.

More Sophisticated Attacks 

Following the recent scamming of Arup, Rob Greig (Arup’s global chief information officer) has been reported as saying : “Like many other businesses around the globe, our operations are subject to regular attacks, including invoice fraud, phishing scams, WhatsApp voice spoofing, and deepfakes.” He noted that “the number and sophistication of these attacks has been rising sharply in recent months”. 

What Does This Mean For Your Business? 

This massive $25 million deepfake scam involving Arup is a reminder of the growing sophistication and severity of digital fraud. Sadly, this incident is not an isolated case but part of a broader trend of increasingly advanced scams leveraging AI. The rapid advancements in AI technology and its wide availability have made it easier for fraudsters to create highly convincing deepfake videos and audio, posing significant risks to businesses of all sizes.

For UK businesses, this incident is a reminder of the urgent need to enhance security measures and verification processes. Traditional methods of authentication, such as emails and video calls, can no longer be solely relied upon. Instead, businesses may want to adopt multi-layered security strategies that include advanced AI-based detection tools, biometric verification, and identity verification protocols. Regular training and awareness programmes for employees may also now be essential to help them recognise and respond to potential threats.

This incident also highlights the critical role of law enforcement and regulatory bodies in combating digital fraud. Enhanced cooperation and information sharing between businesses, cybersecurity experts, and law enforcement agencies are vital to staying ahead of these sophisticated attacks. Implementing stricter regulations on the use and dissemination of AI technology and ensuring that companies have access to the latest detection and prevention tools will be crucial steps in this battle.

The Arup scam demonstrates that even technologically savvy industries are not immune to the threats posed by deepfakes.

Featured Article : Realtime Deepfake Dating Scams

Posted on by Paul Stradling

Here we look at how scammers are now reportedly using face-swapping technology to change their appearance in real-time to conduct video-based romance scams.

Yahoo Boys 

Recently, tech news site ‘Wired’ featured a story about romance scammers dubbed ‘Yahoo Boys,’ a slang term for a Nigeria-based collective of scammers who are now using deepfakes and real-time face-swapping technology so they can take on any appearance in their video feed to the targets of their romance scams. They are also known to be involved in phishing, and other cybercrimes.

Romance Scams 

A romance scam is a type of fraud where someone creates a fake identity to form a relationship with their target, often online, to deceive them into sending money or revealing personal or financial information.

How Big Is the Problem?  

According to the US FBI’s 2023 ‘Internet Crime Report’, the category of ‘confidence fraud/romance’ led to the theft of $652,544,805 from victims (which was actually down by a little over $83 million on the previous year).  This is clearly a significant problem and the real-time component of it will doubtless be factor in making this more prevalent.

How? What Tech Have They Been Using? 

As highlighted by the research of David Maimon, Head of Fraud Insights at SentiLink and a professor at Georgia State University, who has been monitoring the ‘Yahoo Boys’ on Telegram for more than four years, they use phones, laptops and several different types of popular face-swapping software and apps to create their deepfakes.

Also, it’s been noted (by Wired) that the so-called Yahoo Boys post videos of themselves online doing so, often showing their faces in the videos, and the videos and photos of their activities and recruitment are posted across many popular social media channels, including TikTok and Facebook.

Professor Maimon has also noted that the Yahoo Boys started using deepfakes for their scams as far back as 2022, meaning that they have gained quite a lot of experience around using these tools and tactics.

Deepfake Call Types 

It’s also been observed (and highlighted by Wired) that the Yahoo Boys scammers use two different types of live deepfake calls to trick their targets. For example:

Using two phones and a face-swapping app. One phone is used to call the target (via Zoom), using the rear camera to record the screen of the second phone (which is pointing at the scammer’s face) and uses a face-swapping app. In this way, the person’s face the target sees on the real-time video call is completely different from the scammer’s real face.

The second method swaps a laptop for the phone, using a webcam and face-swapping software on the laptop to change the face of the scammer. It’s also been reported that videos made by the scammers of them using this method show that they are able to see their real face displayed alongside their deepfake face although it’s only the deepfake face that’s shown to the target in the video call.

Realistic … and Getting Better

In a LinkedIn post from Professor Maimon, showing an example of one of the scammer’s videos, he notes how “Yahoo boys are getting better using AI tools to bring stolen images of social dating users to live” and that the video example he posted “has piqued my interest due to its remarkably natural head movements, overshadowing the only noticeable flaw—the voice, which could be rectified with relative ease.” 

How To Spot Deepfake (Video Calls) 

On her X feed, Rachel Tobac, who describes herself as a ‘Hacker & CEO at SocialProof Security,’ offers some tips on how to help spot a deepfake video call, based on the latest deepfake calls available.  These are:

– Get the person to stick out their tongue and move it around (tongue will look odd).

– Have the person move their head to the right & left or up & down to a large degree (it will look angular and boxy).

– Ask the person to get close to the camera and turn their head through a wide-angle (see angular boxy side of head).

– Ask the person to add another person next to them in the call and have the original person walk away and come back to see if a deepfake ‘flops-over’ to a second face.

– Look for discoloration around the scalp or circumference of the face (it may look like unblended makeup).

– Look for light flickering in their hair when they move.

Meeting In Person

As noted by contributor ‘Ally A’, to the LinkedIn post about the Yahoo Boys from Matt Burgess of Wired, a key piece of advice to people who may be involved in these kinds of romantic video calls is: “You can’t trust your eyes and ears anymore. If you can’t meet the person you are talking to online IN PERSON within 2-3 weeks of meeting, you have to assume that they are a scammer.” 

AI Advances Helping Scammers

The proliferation of AI technologies and their integration into various applications has inadvertently facilitated the activities of online scammers, including those involved in romance scams. AI-driven tools can now generate realistic and engaging text or images, enabling scammers to create convincing fake profiles and carry out sustained, personalised interactions without much effort – just as the Yahoo Boys have been doing. These sophisticated (but now widely available) tools can help scammers tailor their messages and responses based on the victim’s preferences and responses, making the deceit more believable. As a result, the barrier to entry for conducting such scams is lowered, allowing even those with minimal technical skills to now execute complex and convincing scams, thereby increasing the potential for exploitation and harm to unsuspecting individuals.

How To Protect Yourself 

In addition to Rachel Tobac’s tip for spotting deepfakes (such as those used by the Yahoo Boys), some of the key ways people can protect themselves from falling victim to romance scammers, include:

– Verify profiles. Conduct reverse image searches of profile pictures to check if they appear elsewhere on the internet, which can indicate a stolen image.

– Slow down. Be cautious with individuals who escalate the relationship too quickly or profess love unusually early!

– Keep personal information private. Avoid sharing sensitive personal information such as your address, financial details, or social security number.

– Be very skeptical of requests for money. Be highly suspicious if the person you are communicating with requests money, especially if it is for an emergency or a seemingly urgent matter.

– Use secure communication channels. Stick to the platform’s messaging services and avoid switching to less secure or private communication methods too soon.

– Seek second opinions. Discuss your online relationship with friends or family to gain outside perspectives, especially if something feels off.

– Report suspicious behavior. Report any suspicious profiles or messages to the dating platform and consider filing a complaint with relevant authorities if you suspect a scam.

What Does This Mean For Your Business?

For businesses, understanding the dynamics of the evolving scam landscape, as demonstrated by the techniques employed by the “Yahoo Boys”, is crucial. These scammers, using readily available AI technologies such as deepfakes and real-time face-swapping, underscore a growing trend in cybercrime that leverages cutting-edge technology to exploit vulnerabilities in human psychology, particularly through emotional engagement.

The decentralised nature of these scam networks (where individuals or small groups operate in loose associations while sharing tactics and tools), presents a significant challenge to traditional cybersecurity measures. They operate with a brazen openness, often flaunting their capabilities on social media, which shows a troubling confidence in their ability to evade detection.

The ease of access to AI tools means that the sophistication of scams can evolve as quickly as the technology develops. For businesses, this represents a clear and present danger not just in the form of romance scams targeted at individuals, but as a harbinger of more advanced AI-driven threats that could target companies directly. Phishing scams, impersonation, and business email compromise are just a few examples where similar technologies could be used to deceive employees or manipulate systems for fraudulent purposes.

To safeguard against these threats, businesses need to enhance their defensive strategies by incorporating advanced detection systems that can identify anomalies in communication patterns, authenticate digital identities more robustly, and monitor for signs of emerging threats such as deepfakes. Training employees to recognise and report potential scams is also vital. Creating a culture of security awareness and providing tools to verify information independently can act as a crucial barrier against deception.

Tech News : Microsoft Deepfakes Too Dangerous For Release

Posted on by Paul Stradling

Microsoft says its new VASA-1 AI framework for generating lifelike talking faces of virtual characters is so good that it could easily be misused for impersonating humans and, therefore, Microsoft says it has “no plans” to release any aspect of it until it can be sure it can be used responsibly.

What’s The Problem? 

2024 is an election year in at least 64 countries (including the US, UK, India, and South Africa) and the risk of AI being misused to spread misinformation has grown dramatically.  In the US, for example, the Senate Committee on the Judiciary, Subcommittee on Privacy, Technology, and the Law has held a hearing titled “Oversight of AI: Election Deepfakes”. There is also now widespread recognition of the threats posed by deepfakes and proactive measures are being taken by governments and private sectors to safeguard electoral integrity. AI companies are keenly aware of the risks and have been taking their own measures. For example, Google’s Gemini has been restricted in the kinds of election-related questions that its AI chatbot will return responses to.

Google has also recently (in a blog post) addressed India’s AI concerns as regards its potential impact (deepfakes and misinformation) on what is the world’s largest election. None of the main AI companies have, therefore, wanted to simply release their latest updated generative AI without being seen to test them and include what safeguards they can against misuse. Also, none of the main AI companies are keen to be publicly singled-out as enabling electoral interference.

VASA-1 

Microsoft says its VASA-1 AI can produce lifelike audio-driven talking faces, generated in real-time, all from a single static portrait photo and a speech audio clip.

How Good Is It? 

Microsoft says that its premier model, VASA-1, is “capable of not only producing lip movements that are exquisitely synchronised with the audio, but also capturing a large spectrum of facial nuances and natural head motions that contribute to the perception of authenticity and liveliness.” 

The “core innovations” of VASA-1 include “a holistic facial dynamics and head movement generation model that works in a face latent space, and the development of such an expressive and disentangled face latent space using videos”. 

See some demos of VASA-1 in action here: https://www.microsoft.com/en-us/research/project/vasa-1/

Key Benefits

Microsoft says some of the key benefits of the VASA-1 model that set it apart are:

– Realism and liveliness. The model can produce convincing lip-audio synchronisation, and a large spectrum of expressive facial nuances and natural head motions. It can also handle arbitrary-length audio and stably output seamless talking face videos.

– Controllability of generation. Microsoft says its diffusion model accepts optional signals as conditions, such as main eye gaze direction and head distance, and emotion offsets.

– Out-of-distribution generalisation. In other words, the model can handle photo and audio inputs that weren’t present in its training set, e.g., artistic photos, singing audios, and non-English speech.

– Power of disentanglement. VASA-1’s latent representation disentangles appearance, 3D head pose, and facial dynamics, enabling separate attribute control and editing of the generated content.

– Real-time efficiency. Microsoft says VASA-1 generates video frames of 512×512 size at 45fps in the offline batch processing mode and can support up to 40fps in the online streaming mode with a preceding latency of only 170ms, evaluated on a desktop PC with a single NVIDIA RTX 4090 GPU.

Not Yet 

However, Microsoft says it is holding back the release of VASA-1 pending the addressing of privacy and usage issues, stating that: “we have no plans to release an online demo, API, product, additional implementation details, or any related offerings until we are certain that the technology will be used responsibly and in accordance with proper regulations”. 

What Does This Mean For Your Business?

Given what VASA-1 can do, you’d think Microsoft would be itching to get VASA-1 out there, monetised, and competing with the likes of Google’s Gemini family of models. However, as with Gemini and other generative AI, it may not be fully ready and may have some issues – as Gemini did when it received widespread criticism and had to be worked-on to correct ‘historical inaccuracies’ and woke outputs.

This is also, crucially, an important and busy electoral year globally with governments nervous, trying to introduce legislation and safeguards, and keeping a close eye on AI companies and their products’ potential to cause damaging deepfake and misinformation/disinformation and electoral interference issues, as well as their potential for use in cybercrime. As such, AI companies are queuing up to be seen to be acting as responsibly and ethically as possible, claiming to be holding back and testing every aspect of their products that could be misused – at the same time basically avoiding the eyes of governments and regulators, and potentially bad publicity and penalties.

As some have pointed out, however, it would be difficult for anyone to regulate who uses certain AI models for the right or wrong reasons and that some very sophisticated open source models can be made from source code found on GitHub by those who are determined. All that said, it shouldn’t be forgotten that VASA-1 appears to be very advanced and could offer many benefits and useful value-adding applications, e.g. for personalising emails and other business mass-communication. It remains to be seen how long Microsoft is prepared to wait before making VASA-1 generally available.