Anubis: Protecting Websites from AI Scrapers

Anubis, an open-source program, is helping website operators protect their content from pervasive AI scrapers. Developed by Xe Iaso, it uses “uncaptcha” technology to block unwanted bot traffic, saving resources and safeguarding intellectual property. Organizations like Duke University have reported significant success, blocking millions of daily requests.

The digital landscape, once a vast ocean of freely accessible information, has become a contested battleground.

On one side, the burgeoning titans of artificial intelligence, their algorithms insatiable, voraciously consuming data from every corner of the web.

On the other, the beleaguered website operators, from independent creators to non-profit archives, watching their carefully curated content disappear into the maw of AI models, often without attribution or compensation.

This isn’t just an inconvenience; it’s an existential threat, a silent siege that siphons resources and undermines the very premise of open web contribution.

Indeed, the scale of this digital drain is staggering.

Industry analysis reveals a stark reality: more than half of all web traffic isn’t human.

It’s automated scrapers, a significant portion of which are the crawlers deployed by AI giants.

For website owners, this translates directly into escalating hosting costs, server strain, and the disheartening realization that their intellectual property is being systematically appropriated to train systems that may one day render their original content obsolete.

Enter Xe Iaso, a technical educator with a decidedly practical bent, who has taken up arms in this undeclared digital war.

Frustrated by the rampant exploitation, Iaso engineered Anubis, an open-source program designed to be a self-hosted digital bouncer, stopping AI bots dead in their tracks.

It’s a bold counter-measure, born from the simple, yet profound, desire to protect those who pour their time, effort, and often their own money into creating valuable online resources.

Anubis isn’t some corporate behemoth’s proprietary solution; it’s a grassroots effort, a testament to the power of open-source innovation.

Iaso herself humbly admits it’s “basically the Cloudflare. Are you a bot? page, but self-hostable.” This distinction is crucial.

By allowing organizations to run it on their own servers, Anubis sidesteps the recurring fees associated with commercial services, making robust protection accessible to small organizations, open-source projects, and digital archives – precisely the entities most vulnerable to the resource-intensive tactics of large-scale AI scrapers.

The ingenuity of Anubis lies in its approach.

Operating as a web AI firewall utility, it intercepts all incoming HTTP connections. Before any request can reach the target website, it must first successfully navigate Anubis’s gauntlet.

Iaso, with a touch of dark humor, likens the process to the ancient Egyptian god Anubis, weighing the “soul” of each connection.

The system employs computational puzzles, primarily leveraging JavaScript to perform cryptographic math.

These challenges are trivial for a standard human user operating a modern browser, but they become prohibitively expensive and resource-intensive for bots attempting to operate at the massive scale required for comprehensive data scraping.

Anubis checks for genuine browser behavior, verifying authenticity through JavaScript execution and cookie handling.

When a bot falters, it’s blocked, never touching the core resources of the protected site.

This innovative method positions Anubis as an “uncaptcha.” Unlike traditional CAPTCHAs, which have become increasingly ineffective against sophisticated AI, Anubis operates transparently in the background.

There are no images to decipher, no distorted text to type. For the average user, the experience is seamless, often entirely unnoticed.

The proof-of-work is offloaded to the visitor’s PC or smartphone, ensuring a minimal server-side footprint – typically less than 128 MB of RAM.

It’s a remarkable feat of engineering: distribute the burden, deter the bots, and preserve the user experience.

Iaso is well aware of the hesitations many have regarding JavaScript for security and privacy reasons, and she is already working on a non-JavaScript version, though she acknowledges the significant implementation challenge.

Written in Go and licensed under the permissive MIT License, Anubis embodies the spirit of accessible, community-driven security.

While Anubis’s default configuration is, by Iaso’s own description, a “bit of a nuclear response” – akin to Cloudflare’s “I’m under attack” mode, aggressively challenging everything – this paranoia is by design.

It’s meant to immediately stop the bleeding, allowing administrators to regain control and then incrementally fine-tune their bot policy definitions. This initial “scorched earth” policy might inadvertently block some “good bots” like RSS feed readers or the Internet Archive, but the system allows for explicit whitelisting, and a curated list of known good bots is in development to strike a balance between discoverability and robust protection.

The impact has been immediate and profound. Since its release in January 2025, Anubis has seen over 200,000 downloads.

Organizations like GNOME, FFmpeg, and UNESCO have swiftly adopted it to safeguard their digital infrastructure.

Duke University, a particularly enthusiastic user, reported a staggering success rate: their library systems have blocked approximately 90 percent of unwanted traffic, equating to over 4 million unwanted HTTP requests daily, all while improving service performance with negligible impact on legitimate users.

The battle for the internet’s soul is far from over. As bot developers devise new evasion tactics – from headless browsers to advanced browser fingerprinting – Anubis’s creators are continuously updating the tool, locked in an enduring digital arms race.

The ultimate objective is clear: to maintain an internet that remains openly accessible and usable for humans, while simultaneously making large-scale, abusive bot operations economically unviable.

It’s a monumental task, but with innovative, open-source solutions like Anubis, the fight for a fairer, more sustainable web continues, one blocked bot at a time.

The work is ongoing, and for those who value the integrity of online content, supporting such initiatives is not just an option, but a necessity.