Tag: Web

Sustainability-in-Tech : AI Bots Overtaking Human Web

Posted on by Paul Stradling

AI driven bots are rapidly overtaking humans as the primary consumers of online content, creating growing sustainability concerns around energy use, digital efficiency, and the future structure of the open web.

Report

The latest State of the Bots report from AI bot traffic measurement company TollBit shows a marked acceleration in automated web traffic during the second half of 2025, alongside a measurable decline in human visits. It seems that what was once framed primarily as a debate about AI training data has evolved into a broader structural change, with AI systems now reading the live internet at scale to support search, chat, and information retrieval tools.

Rising Bot Traffic And Declining Human Visits

TollBit’s analysis shows that the ratio of AI bot traffic to human traffic has changed rapidly over a short period. For example, in the first quarter of 2025, the average site monitored by TollBit saw one AI bot visit for every 200 human visits. By the end of the year, that ratio had increased to one AI bot visit for every 31 human visits.

Over the same period, human web traffic declined. Between the third and fourth quarters of 2025 alone, TollBit recorded a 5 per cent fall in human visits across its partner sites. The report stresses that these figures likely understate the true scale of automated activity, as many modern bots are designed to closely mimic human browsing behaviour.

In its findings, TollBit says “from the tests we ran, many of these web scrapers are indistinguishable from human visitors on sites”, adding that the data should be treated as conservative. This increasing difficulty in separating human and automated traffic complicates both measurement and mitigation efforts.

From Training Crawlers To Live Web Retrieval

Earlier concerns around AI and the web focused largely on large scale scraping for model training. While training related crawling continues, TollBit’s data actually shows it is no longer the dominant driver of AI bot activity.

In fact, it seems that training crawler traffic fell by around 15 per cent between the second and fourth quarters of 2025. However, over the same period, traffic from retrieval augmented generation bots increased by 33 per cent.

RAG systems, which fetch live web content to answer user prompts, allow AI tools to provide current answers rather than relying solely on static training data.

This distinction has some important implications. For example, training crawlers typically access content once and store it for offline use. RAG bots, by contrast, return to the same pages repeatedly. TollBit found that in the fourth quarter of 2025, RAG bots made roughly ten page requests for every single page request made by training bots. This repeated access reflects the growing role of AI tools as substitutes for traditional search engines and direct browsing.

The Role Of AI Search Indexing

Alongside RAG bots, AI search indexing activity is expanding rapidly. Indexing crawlers systematically map the web so that RAG systems can locate relevant pages when responding to prompts. TollBit recorded a 59 per cent increase in AI search indexer traffic between the second and fourth quarters of 2025.

This growth seems to show that AI driven search is building out its own parallel infrastructure to support real time information retrieval. While indexing has long been a feature of traditional search engines, the combination of indexing and repeated live retrieval increases the volume of automated traffic moving across the web.

Concentration Of Scraping Activity

TollBit’s data also shows that AI scraping activity is unevenly distributed across providers. For example, OpenAI’s ChatGPT User agent was identified as the most active RAG bot across monitored sites. In the fourth quarter of 2025, it averaged around five times as many scrapes per page as the second most active scraper, attributed to Meta.

Other major contributors include bots operated by Google, Perplexity, Anthropic, and Amazon, each running multiple user agents for training, indexing, and user triggered retrieval. The combined effect is a background layer of automated traffic that now rivals human browsing in scale on many sites.

Which Parts Of The Web Are Most Affected?

It should be noted here that not all content categories seem to be affected equally. For example, TollBit reports that B2B and professional sites, national news outlets, and lifestyle content are among the most heavily scraped. Technology and consumer electronics content experienced the fastest growth in scraping activity, increasing by 107 per cent since the second quarter of 2025.

According to TollBit, the most frequently scraped pages tend to relate to time sensitive topics. In the third quarter of 2025, heavily scraped URLs included political controversies and live sports coverage. By the fourth quarter, entertainment releases and shopping related content, such as streaming series and seasonal buying guides, featured more prominently.

This pattern could be said to reflect how users are increasingly turning to AI tools for up-to-date information, prompting RAG bots to revisit high demand pages repeatedly throughout the day.

The Sustainability Cost Of Repeated Access

From a sustainability perspective, the rise of RAG driven browsing introduces a less visible but growing cost. For example, each automated page request consumes energy across data centres, networks, and supporting infrastructure. When the same content is retrieved repeatedly to support similar prompts, overall energy demand increases significantly.

TollBit, therefore, describes the current environment as inefficient for both publishers and AI developers. AI companies invest heavily in scraping infrastructure, proxy services, and evasion techniques, while publishers spend increasing sums on defensive technologies. This duplication of effort results in higher processing and energy use, alongside increased indirect emissions.

In fact, the report notes that advanced scraping services can charge more than 22 dollars per 1,000 pages retrieved. At the scale required to support popular consumer AI applications, data acquisition costs alone can reach tens of millions of dollars per year. These financial costs sit alongside rising electricity demand in data centres, which sustainability researchers already identify as a growing contributor to global emissions.

Robots Txt And Escalating Inefficiency

Existing mechanisms for controlling automated access seem to have proven ineffective. For example, in the fourth quarter of 2025, around 30 per cent of AI bot scrapes recorded by TollBit did not actually comply with robots.txt permissions. In categories such as deals and shopping, non-permitted scrapes exceeded permitted ones by a factor of four.

OpenAI’s ChatGPT User bot showed the highest rate of non-compliance among major bots, accessing blocked content in 42 per cent of cases. TollBit argues that this environment encourages increasingly sophisticated evasion strategies, including IP rotation, user agent spoofing, and cloud based headless browsers.

Each layer of evasion and detection adds computational overhead. Bots expend more resources to appear human, while websites consume more resources attempting to identify and block them. From an environmental standpoint, this escalation increases energy use without delivering proportional value to end users.

Low Referral Traffic And Structural Implications

The sustainability issue is closely tied to the economics of online publishing. TollBit reports that referral traffic from AI applications remains extremely low and continues to decline. Average click through rates from AI tools actually fell from 0.8 per cent in the second quarter of 2025 to 0.27 per cent by the end of the year.

Even websites with direct licensing agreements saw some pretty sharp declines. For example, click through rates for sites with one-to-one AI deals fell from 8.8 per cent early in 2025 to 1.33 per cent in the fourth quarter. This indicates that licensing arrangements alone are not insulating publishers from reduced human traffic.

The result, therefore, appears to be a system in which machines read and reuse content at scale, while fewer people visit the original sources. For example, TollBit’s report states that “AI traffic will continue to surge and replace direct human visitors to sites”, pointing to a future in which automated systems become the primary readers of the internet.

The data suggests that this transition is already underway, with some significant implications for sustainability, digital infrastructure, and the long-term viability of the content ecosystem that AI systems depend on.

What Does This Mean For Your Organisation?

The picture emerging from TollBit’s data seems to be one of structural change rather than a short-term disruption, where AI systems are no longer just indexing the web or training on it in the background. In fact, it seems they are now repeatedly consuming live content at scale, with clear consequences for energy use, infrastructure efficiency, and the sustainability of the wider digital ecosystem. Without changes to how AI systems access content, the current pattern risks locking in higher energy demand and escalating inefficiencies across both AI development and online publishing.

For UK businesses, this trend has practical implications on several fronts. For example, organisations increasingly relying on AI tools for research, search, and decision support are indirectly contributing to rising digital energy use and associated emissions. At the same time, UK publishers, professional services firms, and content driven businesses face growing operational costs from defending their websites against automated access, while seeing diminishing human engagement in return. These pressures sit alongside wider regulatory and sustainability expectations, particularly as UK businesses are required to demonstrate progress on energy efficiency, emissions reporting, and responsible technology use.

For AI developers, publishers, regulators, and end users, the data shows that the current scrape and block dynamic appears inefficient, costly, and environmentally counterproductive. If AI systems are to become permanent fixtures in how information is accessed, it looks as though the underlying mechanics of content access will need to evolve in a way that supports sustainability, fair value exchange, and long-term viability. Without that recalibration, the growth of AI driven web consumption risks undermining both the digital economy it depends on and the sustainability goals many organisations are now expected to meet.

Company Check – Google.co.uk Phased Out

Posted on by Paul Stradling

Google is retiring all country-specific search domains, meaning users who try to visit sites like google.co.uk will soon be automatically redirected to google.com instead.

Unified Search Experience

Google is ending its long-running use of country-specific domain names like google.co.uk, redirecting all users to a single global homepage, i.e. google.com. The change, already rolling out, marks a decisive step in Google’s ongoing shift towards a unified, location-aware search experience that doesn’t rely on domain suffixes to deliver local content.

Why Is Google Making This Change Now?

Although announced back in April 2025, Google first restructured its approach to localised results much further back in 2017. At that time, the company moved away from delivering search results based on which domain you typed (such as google.co.uk or google.com.au) and instead began using the physical location of a user to determine what they saw.

Google explained this at the time by noting that “one in five searches is location-related,” suggesting that using a device’s GPS or IP address was a more accurate way to serve local content than relying on top-level domains.

Fast-forward to now, and Google believes its localisation technology has advanced far enough to render ccTLDs (country code top-level domains) obsolete. That includes not just .co.uk for the UK, but also .com.br for Brazil, .co.in for India, .fr for France, and so on.

In a statement issued on 15 April 2025, Google said: “Because of this improvement, country-level domains are no longer necessary. We’ll begin redirecting traffic from these ccTLDs to google.com to streamline people’s experience on Search.”

What Exactly Will Happen, and When?

Google says it’s rolling out the change gradually “over the coming months.” Users who continue typing local addresses like google.co.uk into their browser will automatically be redirected to google.com instead.

What’s important to note is that this redirection is cosmetic and, as such, it won’t alter the substance of search results. Google is keen to reassure users that location-based relevance will still be maintained, saying: “This update will change what people see in their browser address bar, but it won’t affect the way Search works.”

It should be noted, however, that in some cases, users may be asked to log back into their Google account or re-enter search preferences such as language, region, or safe search filters. These prompts are part of the transition and are expected to be minimal.

What About Businesses and Search Professionals?

While most casual users may hardly notice the difference, the change is likely to have more significant implications for businesses, advertisers, and SEO professionals.

For years, digital marketers and local businesses relied on country-specific domains as a signal for local intent. Seeing .co.uk in the address bar reassured UK users they were getting localised content. With that visual cue gone, it seems that businesses may need to work harder to communicate relevance to their audience.

Some SEO consultants have already raised concerns, that ccTLDs have long played a psychological role in establishing trust and a sense of local identity and that removing them could take away a small but meaningful visual cue that helps users quickly judge whether a result is relevant to their region.

On the technical side, it’s more of a mixed bag. For example, consolidating everything under google.com may mean less domain fragmentation and a cleaner search experience, but some businesses fear they’ll have less control over regional visibility.

That said, Google’s localisation signals are now based on far more than just the domain name. For example, language, browser settings, search history, and even device-level data play a role. From a technical standpoint, this change may encourage businesses to invest more in structured data, content localisation, and region-specific marketing rather than relying on domain-level cues.

Will It Save Google Money?

Although cost-cutting wasn’t mentioned in Google’s official statement, it’s difficult to ignore the potential operational efficiencies behind this move.

It’s likely that maintaining dozens of ccTLDs, each with separate infrastructure, legal requirements, and occasional localised content, is expensive. As Google streamlines its services across all its products, it’s plausible this change is part of a broader effort to reduce complexity and overheads.

Keeping a multitude of domains also introduces additional security considerations and potential legal complications in different jurisdictions. Consolidating to google.com, therefore, offers a more scalable, consistent platform from both a technical and administrative point of view.

A Brief History of ccTLDs on Google

Google’s use of country-specific domains stretches back to its early international expansion in the early 2000s. Back then, separate ccTLDs made sense as they helped users access locally relevant content and gave governments and regulators some measure of oversight.

In fact, ccTLDs were once a key part of Google’s global strategy. For example, there was google.ca for Canada, google.co.jp for Japan, google.co.za for South Africa (the list goes on). For over a decade, typing a country-specific address was the main way users navigated to “their version” of Google.

However, by 2017, the writing was already on the wall. As mobile use exploded and GPS-based location detection improved, ccTLDs became more of a legacy feature than a critical component of localisation. By delivering results based on physical location rather than the domain used, Google effectively decoupled the URL from the search experience.

As Google put it back then: “Typing the relevant ccTLD in your browser will no longer bring you to the various country services—this preference should be managed directly in settings.”

Now, eight years later, Google’s retiring those domains for good.

How Can Users Still Control Their Search Experience?

With the ccTLD route disappearing, users who want to customise their search region still have some options. For example, Google recommends visiting the settings on google.com, then selecting ‘Settings’ (bottom-right corner), ‘Search settings’, and then ‘Language and region’.

From there, users can select their ‘Results Region’ and confirm their preferences. It’s not quite as effortless as typing .co.uk into the address bar, but it gives users some control nonetheless. It’s worth noting that this could also become a more critical step for people who travel frequently or use VPNs, where physical location and desired results don’t always align.

What Does This Mean For Your Business?

Google’s move to retire country-specific domains marks the end of an era, but not necessarily a dramatic shift in day-to-day use. For the vast majority of users, the redirection to google.com will be seamless, with local results continuing to surface just as they did before. In that sense, the change is largely symbolic: a visible reminder of how far the technology has come since the days when a .co.uk domain was essential for getting UK-relevant content.

Even so, the impact will be felt in some corners. For businesses and SEO professionals who have built strategies around ccTLDs, there’s now a need to refocus efforts. Visibility in local search will depend less on the domain in the address bar and more on how well a business optimises its content for a specific region using technical signals and structured data. That may be more work in the short term, but it could ultimately lead to a more level playing field, particularly for smaller businesses that don’t operate country-specific sites but still want to compete in local markets.

For UK businesses in particular, there’s a communications challenge. Losing the .co.uk cue means finding other ways to demonstrate relevance and trust to a local audience. Whether through clear regional language, local contact details, or targeted content, companies will need to be intentional about showing that they’re rooted in the UK market. It may also prompt more attention to things like Google Business Profiles, location-based advertising, and hyperlocal content strategies.

For Google, this is as much about simplification as it is about modernisation. By consolidating its domains, the company reduces redundancy, eases infrastructure demands, and likely saves money, all while continuing to meet legal obligations in different countries. It also fits neatly with Google’s wider goal of delivering a consistent user experience across its ecosystem, from Search to Maps to AI-powered features.

Ultimately, this change reflects how the web, and the way we use it, has evolved. Search is no longer bound by the old rules of geography and domain suffixes. It’s now driven by data, context, and personalisation. This means that while the disappearance of google.co.uk from the browser bar may feel like a nostalgic loss for some, the mechanics of search will continue to evolve around us, often invisibly, and usually with far-reaching implications.

Tech Tip – Secure Your Web Browser in Under 5 Minutes

Posted on by Paul Stradling

Think your antivirus is enough? If your browser isn’t secure, your data, and even your logins, could be at risk. Luckily, tightening things up takes just a few minutes. Here’s how:

How to lock things down (Chrome example):

– Click the three dots in the top right, then go to Settings.
– Select Privacy and Security.
– Turn on Enhanced Protection under Safe Browsing — this gives you faster, smarter protection from phishing, dodgy sites, and harmful downloads.
– Go to Cookies and other site data, and choose Block third-party cookies for more private browsing.
– Scroll to Site Settings, then:
– Block pop-ups (unless you really need them).
– Disable camera and microphone access unless needed.
– Check permissions for location, notifications, and clipboard, and switch off anything that looks unnecessary.

Bonus steps:

– Install a reputable ad blocker (e.g. AdGuard – there are many others).
– Use a password manager instead of letting your browser save logins.
– Make sure automatic updates are enabled so you’re always running the latest security fixes.

Pro-Tip: These steps apply to most major browsers – Firefox, Edge and Brave all have similar privacy and security menus. Just search for Privacy settings in their menus and follow the same logic.