Got bad news about this.
Over the weekend, i was able to compile a good banlist against 2 of 3 Chinese networks hammering our site.
I thought this would get us ahead.
Then i saw the Brazilian traffic flood over last weekend..
Xenforo's site ( using cloudflare ) got hit first and i saw it rise to 16k bots.
Ours got hit next and went up to 20k bots and the flood lasted 2x longer.
Even though my system averages as good as cloudflare.. neither technologies provide the level of protection against what the internet can dish out lately.
The bots are not doing anything productive and are just slamming 2 URLs that don't provide them information.
The problem: the IP addresses are too numerous ( 10,000's of them ) and each one makes 2-4 very slow hits.. so this is impossible to defend against with even some sophisticated rate limiting.
Our site held up fine and i think the reason was the recent round of TCP/IP tuning.
The only damage was to my weekend peace.
A major cost problem exists: we're on AWS, and bandwidth is expensive at a couple TB per month.
95% of this bandwidth cost is bots.
I looked over things and it appears the AI scraper bots are increasing our monthly cost of operation by 275 USD/month in bandwidth costs alone.
On top of the time i spend checking the logs, updating permanent banlists, etc.
Which means this site is ~4x more expensive to run than since the beginning of AI bot armageddon.
Thoght process on what to do:
Goals:
- dramatically reduce costs
- make the sysadmin's life much easier
- make the lives of people running internet strip mining operations very hard
Chosen route:
- move to Hetzner ( after testing for speed ) for much lower costs
- test and possibly implement anubis as a 2nd layer defense to make these people's lives harder
- finish the next generation, much smarter protection mechanism and replace anubis with it.
I'll update this thread once we get cracking.
Over the weekend, i was able to compile a good banlist against 2 of 3 Chinese networks hammering our site.
I thought this would get us ahead.
Then i saw the Brazilian traffic flood over last weekend..
Xenforo's site ( using cloudflare ) got hit first and i saw it rise to 16k bots.
Ours got hit next and went up to 20k bots and the flood lasted 2x longer.
Even though my system averages as good as cloudflare.. neither technologies provide the level of protection against what the internet can dish out lately.
The bots are not doing anything productive and are just slamming 2 URLs that don't provide them information.
The problem: the IP addresses are too numerous ( 10,000's of them ) and each one makes 2-4 very slow hits.. so this is impossible to defend against with even some sophisticated rate limiting.
Our site held up fine and i think the reason was the recent round of TCP/IP tuning.
The only damage was to my weekend peace.
A major cost problem exists: we're on AWS, and bandwidth is expensive at a couple TB per month.
95% of this bandwidth cost is bots.
I looked over things and it appears the AI scraper bots are increasing our monthly cost of operation by 275 USD/month in bandwidth costs alone.
On top of the time i spend checking the logs, updating permanent banlists, etc.
Which means this site is ~4x more expensive to run than since the beginning of AI bot armageddon.
Thoght process on what to do:
| Option | Pros | Cons | Makes Money Sense | Time needed to make change |
| Move from AWS to Hetzner Dedicated | We can shave server costs by 80%. No longer need to worry about the Bandwidth cost, just the problem that our data is being heisted. Good long term move if the hell is going to persist / ramp up ( most likely ) | - Can no longer simply export a Virtual Machine to create a perfectly accurate development environment, now this needs to be done by hand or automated ( although this only needs to be done every 2-4 years ) - Unmanaged bare hardware can be less reliable than cloud hosting, don't like that i possibly have to respond to hardware SHTF - No selective upgrade/downgrade of server - Located in Germany so Americans in the middle of the USA will see +150ms to response time | Totally | 4 days ( move, plan, automate ) |
| Move from AWS to Hetzner Dedicated Cloud US | Same as above but 60% | - Web server will be much less snappy, you'll notice it during heavy searches. | Totally, but with compromises | 4 days ( move, plan, automate ) |
| Submit to cloudflare | Better at handling mass insanity Should require less intervention | - Amberwolf can't see the site - Reduction in uptime ( +1-3 days/year ) - Still needs to be messed with from time to time - Bandwidth reduction may only be ~25% - Contributes to centralization of the internet | Yes, but too many compromises | 1-3 days? |
| Adopt Anubis | Approximately equal to cloudflare, more effective as more website operators use it | - Rapidly improving but not mature yet - $50/mo for unbranded version - From a reduction of operations cost perspective, it needs to pay for itself | Very possible | 1-3 days? |
| Develop my protection system | Have your cake / eat it too solution that can be catered to ES' needs | - In what free time? i will be slow to complete this. - Small possibility that it won't work well | Barely ( development cost can be split between 5 parties ) | ~30 days / 5 = 6 days/party |
| Do nothing | Doesn't cost developer time | Server and bandwidth cost continues to increase | No | 0 days |
Goals:
- dramatically reduce costs
- make the sysadmin's life much easier
- make the lives of people running internet strip mining operations very hard
Chosen route:
- move to Hetzner ( after testing for speed ) for much lower costs
- test and possibly implement anubis as a 2nd layer defense to make these people's lives harder
- finish the next generation, much smarter protection mechanism and replace anubis with it.
I'll update this thread once we get cracking.
Last edited:
