Anubis: Open-Source AI Firewall to Block Scraper Bots
▼ Summary
– Anubis is an open-source tool that protects websites by adding computational friction to deter automated scraping while allowing human access.
– It functions as a gatekeeper, requiring a client’s browser to complete a small proof-of-work challenge before granting access to the web service.
– The tool operates as a reverse proxy, allowing easy integration without rewriting existing applications through simple configuration files.
– It provides operators with control and transparency, letting them tune challenge difficulty and exempt trusted traffic like health checks.
– Its protection is economic, raising the cost of bulk requests without using behavioral analysis, making it a focused solution for operational scraping problems.
Protecting a website from automated scraping and excessive bot traffic is a constant challenge for administrators who need to keep content accessible to legitimate users. Anubis is an open-source tool that addresses this by introducing a layer of computational friction before serving requests. Maintained by TecharoHQ, this project offers a practical solution for operators aiming to deter large-scale automated data collection without disrupting human visitors.
The tool functions as a gatekeeper positioned in front of a web service. When a client attempts to connect, Anubis can require the browser to solve a small proof-of-work challenge before granting access. This concept is adapted from older anti-spam techniques, where imposing a minor computational cost was negligible for individuals but prohibitively expensive for mass operations. In a web context, this effectively raises the economic barrier for bulk scraping activities while remaining virtually unnoticed by regular site visitors.
The challenge itself is delivered via JavaScript and executes within the user’s browser. Upon successful completion, the browser obtains a token that permits subsequent requests to proceed without repeating the task. Site administrators have the flexibility to adjust the difficulty of the challenge, allowing them to fine-tune the balance between robust protection and a seamless user experience based on their specific traffic patterns and risk assessment.
Deployed as a reverse proxy, Anubis sits between end-users and the origin server, forwarding requests only after challenge conditions are satisfied. This architecture enables integration without the need to modify existing applications. Configuration is managed through straightforward files that specify which routes or clients should receive challenges and how those challenges are issued.
A key principle of the project is providing operators with control and transparency. They can define precise rules for when challenges are applied and when traffic should bypass them entirely. This capability supports essential operational needs, such as permitting uninterrupted access for trusted services, health check monitors, or internal users. Comprehensive logging and metrics offer clear visibility into how frequently challenges are triggered and how clients interact with them.
The development of Anubis was driven by the real-world problem of sustained automated scraping placing heavy loads on community-run websites. The maintainers position it as a focused, practical response to a specific operational issue rather than a broad-spectrum security platform. This narrow focus is evident in its design, which avoids feature bloat and remains dedicated to its core function.
From a security standpoint, Anubis does not rely on identifying bots through behavioral analysis or reputation databases. Its protection model is fundamentally economic: each request incurs a small cost, making large-scale automation financially burdensome over time. This approach results in predictable, understandable controls for administrators seeking a straightforward defensive mechanism.
The complete project is freely available on GitHub for anyone to use, modify, or contribute to.
(Source: HelpNet Security)
