ES 2.0 Forum updates and tribulations

[neptronix]

My theory is that we are on the deflationary side from the notable covid ebike/bicycle/sporting equipment in general/hobby bubble. Combined with apparent maturity in the DIY ebike hobbyist fabrication environment (most processes have been tried and documented, no new paradigm-shifting products being discovered or hitting the market). And the entry level prebuilt ebikes are hitting an attractively low buy-in price.

Unfortunately yes.

One thing we haven't done for the public is speed up the learning process. A forum is temporal format. It takes dedication to read ES for months to get up to speed.
That doesn't compete well with a, lately, cheaper prebuilt solution. That's the new way to get your feet wet.

Here's another factor: Chinese stuff is a lot more expensive in the USA, not sure about elsewhere in the west.
Just bad economic conditions.

The right call is to fix what you got.... and hopefully you bought quality!

We are still in the runup to a possible (probable?) worsening petroleum crunch, which would drive more general public interest back to ebikes, as covid did.

Man, i have no idea how that's going to play out. it's so unpredictable. But usually when the USA decides to interject in the middle east, they stay there for a while. EU cares about green things but they're busy busting even 250w bikes because they have a throttle.

I too have been seeing fresh new reliance on Cloudflare on numerous websites. Monoculture seems like a fitting label, with its inherent risks.

Yes, everyone is turning on 'under attack mode' because the level of protection they are offering on the < 4 digits $ per month tiers is inadequate.

What's limited us on adopting advanced protection is that we need deep customizability.

The algorithm to defend against this has been bouncing around in my head for a year. Just didn't have the right technology to do it fast and simple until last week's research.

Not that i needed more work. But it's my job to do what needs to be done.

 

[neptronix]

Speak of the devil.
We just had a 5 minute gigablast of AI scraper traffic yesterday. They got a high score of almost 70k AI scrapers.





I happened to be watching the server at the time.
It survived without interruption only thanks to the high scale configuration from the last rounds of battle, and the fact that the forum software ( Xenforo ) is pretty efficient.
This was partially due to a chink in the armor which has been double reinforced now.
Another cloudflare + Xenforo user reports getting slammed by 85k bots. That would have probably made the site stutter.

I have been working on a prototype for a v2 of my protection system over the last two weeks after finding a way to make it handle at least in the 100,000 hits per second range on a 8 core system.

Which is the internet traffic equivalent of a category 4 hurricane ( very few servers can handle it )



I did the math and looked over the traffic from the attack. My v2 would have done 40% better with just 1 of 7 layers of protection enabled. Another layer is said to add ~50%. So it probably works well.

v2 is very difficult to design because of performance. But long ago i read about how Facebook made PHP fast enough to serve a billion users in the 2010's. I tried their technology after a few failures and was able to process 67k requests/second on a very mid 4 core processor. So i am convinced i cracked the problem.

I have all the hard parts of the design for v2 finished except for one edge case i noticed with this particular attack.
Will start working on the full proof of concept next week.

In the meantime we have a forum plugin that implements 1 layer of my v2 system. If the scrapers want to cause trouble over the weekend, they have a surprise waiting for them in the interim.



Hope you are enjoying the site. We go through great lengths to keep the machine humming, but the internet is a bee-yatch these days!

 

[harrisonpatm]

Speak of the devil.
We just had a 5 minute gigablast of AI scraper traffic yesterday. They got a high score of almost 70k AI scrapers.

View attachment 389481

View attachment 389493

I happened to be watching the server at the time.
It survived without interruption only thanks to the high scale configuration from the last rounds of battle, and the fact that the forum software ( Xenforo ) is pretty efficient.
This was partially due to a chink in the armor which has been double reinforced now.
Another cloudflare + Xenforo user reports getting slammed by 85k bots. That would have probably made the site stutter.

I have been working on a prototype for a v2 of my protection system over the last two weeks after finding a way to make it handle at least in the 100,000 hits per second range on a 8 core system.

Which is the internet traffic equivalent of a category 4 hurricane ( very few servers can handle it )

View attachment 389494

I did the math and looked over the traffic from the attack. My v2 would have done 40% better with just 1 of 7 layers of protection enabled. Another layer is said to add ~50%. So it probably works well.

v2 is very difficult to design because of performance. But long ago i read about how Facebook made PHP fast enough to serve a billion users in the 2010's. I tried their technology after a few failures and was able to process 67k requests/second on a very mid 4 core processor. So i am convinced i cracked the problem.

I have all the hard parts of the design for v2 finished except for one edge case i noticed with this particular attack.
Will start working on the full proof of concept next week.

In the meantime we have a forum plugin that implements 1 layer of my v2 system. If the scrapers want to cause trouble over the weekend, they have a surprise waiting for them in the interim.

View attachment 389491

Hope you are enjoying the site. We go through great lengths to keep the machine humming, but the internet is a bee-yatch these days!

Its rather interesting reading about your actions, from a lay standpoint

 

[neptronix]

Glad someone enjoys reading it!

I figured out the edge case in my v2 protection system over the weekend and figured out how to lower the RAM requirements while i was at it.

I watched Xenforo.com yesterday have a cloudflare break-through with 190k guests versus our record of 70k. Bit chilling.


We've been sitting on an updated version of our image recompressor for a while.
To close out ES 2.0, we did a test run on it on our development server, and the results:

Original Size: 142.43G
Current Size: 54.78G
================================
Size Reduction of: 61.56%
Files processed: 575,931

Very good result as expected.

We have 2 transformations that fix pre-2023 era content:
- add hyperlinks into the content ( phpbb did this automatically )
- youtube link fix ( many types of phpbb youtube tags broke )

These need a 200 sample lookover of both along with the results of the image processor.
Me and the Kahuna are both busy for years, so we're hiring a tester to do get the review part done soon.

 

[E-HP]

Glad someone enjoys reading it!

I figured out the edge case in my v2 protection system over the weekend and figured out how to lower the RAM requirements while i was at it.

I watched Xenforo.com yesterday have a cloudflare break-through with 190k guests versus our record of 70k. Bit chilling.

I was a developer for a while before switching to the business/client side. IT assigned a few developers to my project. At first they didn't like it, because I could tell when they were padding their estimates, etc., but then they like it because I could understand the challenges and could help with solutions. But, my demands were pretty high, because I knew what was possible, and they'd walk out my office shaking their heads and moaning. But, when they'd come back later with a solution, they were super excited, I think because they impressed themselves by what they had done. I miss being on both ends of that, but seeing the sense of accomplishment is almost as rewarding as being the one that did the work.

Keep protecting the site! It's for a noble cause.

 

[99t4]

I watched Xenforo.com yesterday have a cloudflare break-through with 190k guests versus our record of 70k. Bit chilling.

Wow! Was that 70k from legitimate activity or AI scraping?

 

[neptronix]

I was a developer for a while before switching to the business/client side. IT assigned a few developers to my project. At first they didn't like it, because I could tell when they were padding their estimates, etc., but then they like it because I could understand the challenges and could help with solutions. But, my demands were pretty high, because I knew what was possible, and they'd walk out my office shaking their heads and moaning. But, when they'd come back later with a solution, they were super excited, I think because they impressed themselves by what they had done. I miss being on both ends of that, but seeing the sense of accomplishment is almost as rewarding as being the one that did the work.

Didn't know this about you. You're describing my life somewhat

Keep protecting the site! It's for a noble cause.

Ya welcome. It's good to have a hard case in my server fleet, i've learned some strong linux fu on it, and it's worth it because i love the community.

Wow! Was that 70k from legitimate activity or AI scraping?

The historic guest count ( pre-2024 ) for this site is in the low hundreds, so easily 99% of that is scrapers.

 

[neptronix]

Bots had the site down for about 15 minutes this AM because they hit a new limit i haven't seen them hit before.
I 4x'd that limit and hopefully we have that mole whacked.

 

[neptronix]

Some recent news.

Construction of my v2 security/scraper defender mechanism is underway.
A fellow Xenforo administrator jerry rigged two plugins that utilize 2 of my 8 layers, and the result was better than cloudflare.
How lucky to have partial confirmation of this idea before building the first prototype!

( we would be using this setup today if both these plugins didn't enormously slow down the site or crash during an attack. At our scale, that's going to happen. We found out that one of them is vibe coded and getting a lot of complaints from users regarding performance.. so.. not a great idea )

Image processor testing is holding us up from making the big server move.
This always takes a long time because each time you run it on the ES dataset, it's going to take over a day to process the whole set.
Both painful to work on, and super technically impressive at the same time.

I can't wait to have the infra/bot control done for a while so we can get to improving the forum format.

Happy riding to you all

 

[neptronix]

Sup.
Protection is holding sturdy and we're about to crack a record of 30k banned IPs.
I developed a better vantage point to spot large commercial networks very quick. I plan to release that as open source.
Eventually that identification will be fully automated.



I combined some recent memory and performance optimizations i developed on other systems and cut the memory footprint in half. This means we can shrink down to an even smaller server.

Hetzner turns out to be a bust. Way too much hassle on payments, and our staging site was down for 5 hours due to their issue.
There is such a thing as too cheap.

Looks like we're moving to Vultr. Bandwidth ( our main cost ) is still 90% less and i've had 6 years of good experiences with that one.

I look forward to not having to spend any time managing the size of the bill, having more funding go to feature development, and sleeping easier.

Happy biking!

 

[neptronix]

Well howdy!

The big Kahuna looks like he's got ES ready to pump from one server to another in 1 motion. We noticed some erratic behavior on the new server but it just turns out to be a difference in how Vultr vs. AWS sets up their machines. Phew!

The image processor needs to run on *.* for another 2 days, then we can enter the testing phase starting friday or monday.
I'd guess we're at the 90-95% done mark.

The known unknown is how the new Anubis protection works at scale. But it can be turned off and we can revert to the existing system that works well enough. Failure is an option

I'm so excited to close out the list of what makes up ES 2.0, because 2.4 is the fun part where we get to focus on functionality over what was mostly infrastructure, AI scraper control, getting to PHPBB feature parity, modernizing some things, fixing what broke in the translation, and cost cutting work.

The yogurt on top is eaten and we're on to the fruit on the bottom part. For real.


Speaking of AI, at our dev shop, we should have the hardware to run a full quality version of the new DeepSeek Flash 0731 in a few days. Deepseek is known for their optimization genius... and their latest 'small' model is totally nuts..


Previously, we are running the model on the right.. which is still best in it's class for it's size.
One note.. Deepseek has not even released their updated larger model yet.. that's going to be very disruptive.



To test the 'flash' version, i found a quantization of the model that loses 15% of it's bits, and ghetto rigged a NVME to PCIE adapter to add 12GB of vram to the rig. So, total of 140GB VRAM.



I started my tests from a basic prompt: 'make me a shooter game', then molded from there, to see how far and how long it could remain coherent and not goof anything up while i kept changing my mind and making it more complex. I provided no technical hints and purposefully tried to trip it up.

Many hours and 153k tokens ( max limit with current hardware ) later, i shaped it into a medium to low difficulty late 90's 'bullet hell' type shooter with good mechanics, difficulty balance, and progression... but i ran out of VRAM before i could perfect it, so it has rough edges but is otherwise actually a good game.

Feels like i barely scratched the surface of what this is capable of.



The game is attached in a .zip, and fully contained in a 72kb html file. The source code looks very nice, and performance is good.
3 other people told me they enjoyed it.

6 levels and 3 bosses.
ctrl key = shoot bomb
hold space = shoot
arrow keys = move


Quality wise, I can confirm it's ~2x better than what we're running, despite some brain damage from quantization and using a reasoning setting of 'high' instead of 'max' <-- substantially better but too slow to use because the small GPU is choking the setup!

I have another GPU on the way and we are going to run this beast right.


Steve Jobs said 'the computer is a bicycle for the mind'.

From a programming perspective, AI, if you wield it right, can become an electric motor to the bicycle for the mind... which, if skillfully and tastefully utilized, we can use to dream much bigger than our budget here at ES... so it's great timing as we're about to dig into the fruit.

We look forward to understanding what this bad boy can do and what should still be done with the manual plow vs not.
Generally we want to enhance humans and not replace them. You still need systems thinkers and designers to produce good output. Otherwise, you get slop.

Even more lucky for us is that the provider of the forum software made some 'AI skills' that this model should be smart enough to use. So this electric bicycle for the mind already comes with a good tune, extra heatsinks, and ferrofluid



More on this AI business, and the move, later.

Happy riding!

 

[neptronix]

Yo!

Finally the Xenforo team said something about the new version of the software and posted a TODO list indicating that they're in the wrapping up part.



Finally some communication after over a year!

On our end:
- The testing for our server migration + content transformations is underway.
- Initial deployment of the new AI model has very good intelligence, but kinda underwhelming performance. Tuning of the new AI hardware and software is 50% done. The next tuning round should yield a 10-50% speedup, which should be adequate to start putting the hardware to work at ES.

Happy riding!