ES got hammered ( and lived )

[neptronix]

Our server yesterday saw what looked like a year's worth of traffic over a few hr period.

I got an alarm that the CPU was at 100% for longer than 4 minutes and went to work investigating what was going on. At first, i thought it was a DDOS. Took me a bit to figure out what was going on. I have never seen a traffic spike like this in 7 years at ES.




Yeah, at one point, the server was outputting gigabyte a second. Absolutely nuts.



You can also see in little dips in the CPU graphs, the small adjustments i made to try to optimize for traffic conditions, before eventually going from a 2 core, to a 4 core, then a 8 core machine to handle the traffic. We were down for about 4 minutes in total, only because i scaled the server to provide faster page output - not because it was about to crash.

Well, it turns out that one of our members here submitted their build thread to ycombinator's "hacker news" site., and his post was #1 for around 8 hours.

I think it's cool that our site got so much attention in the end. I do wonder if it's an indicator of the interest in ebikes the days ^_^

This was also a great test of our server setup's mettle. And a good reason to think about implementing a really killer cache mechanism so that the next server flogging is a real non-issue.

 

[nicobie]

That's interesting. I'll have to check out that website.

 

[marty]

Was having some computer confusion. Logged out for a bit. Looked at this while logged out. See:

You do not have the required permissions to view the files attached to this post.

See screen shot.

Think forum would better if the world is allowed to see pictures.

For a while https://www.mcmaster.com/ would not let me view drawings of stuff unless I created a account and logged in. Now I can. Wonder why they did that? Wonder if their servers could not handle to many people looking at and downloading CAD drawings and stuff?

 

[neptronix]

This exists as a standard feature of phpbb for a long time.
Remember that at one point, we were transmitting 1 gb/second... OF HTML

Now, you serve the images and you're all the sudden in the terabytes/sec range and we need an entirely different hosting plan and network configuration. What happens in an event like this when we don't have that infrastructure is that nobody can load ES. And possibly, the provider pulls the plug because they also think they're seeing a DDOS also.

I would be open to seeing if we could serve images to unregistered users, but they'd have to be watermarked low quality versions ( maybe some text that says 'come join us at endless-sphere.com' ) in order to deter hotlinking, and slow the flow of traffic during mass events like these, but still give visitors a sense of what's on here.

My image processing system, imagetender, could produce these watermarked, 'vieweable enough' images while also crunching through what's been uploaded, with a non-hurculean effort. So it's possible to at least generate these images.
Xenforo also may have a system for this which would be easy to hijack.

I should certainly look into this once we're settled into xenforo.

 

[marty]

https://www.phpbb.com/community/viewtopic.php?p=11055565#p11055565

Re: Guests can't view images

Post by JimA » Thu Oct 22, 2009 2:41 pm
Go to your ACP, Permissions tab, Groups' forum permissions.
Select the Guests group, and the forums you want this permission to apply for.
Click "Advanced permissions" -> Misc

Set "Can download files" to Yes.
Jim Mossing Holsteyn - Former Community Team Leader
Knowledge Base | Documentation | Board rules

If you're having any questions about the rules/customs of this website, feel free to drop me a PM.
Top

Not sure if this would let all people on the planet look at pictures?

 

[neptronix]

It would, and that's why we can't use it.
We will figure out how to turn the 'terabytes per second' problem before such a setting is ever turned on.

 

[calab]

What does 200,000 network packets or 1TB actually mean in terms of people actually using the es forum?

 

[neptronix]

Okay i'm gonna try to keep it simple as possible here..

In this scenario where we're serving full size images, we'd need about 1000gb/sec out, and AWS maxes out at 50gb/sec output even in the super expensive hosting tiers.

With a 95% shortfall in bandwidth.. our site absolutely goes down. You have a 5% chance to view a completed webpage. If you are one of the lucky ones that can view the site, each page loads in minutes, not seconds, if at all.

This is bad.

In the scenario where everyone unregistered gets potato quality images, we're only missing 50% of the bandwidth we need for this megaspike, so everyone will get a completed page... but, in seconds instead of the usual milliseconds.

Unless we also get a crapload of signups. Then we're screwed. :lol:

 

[marty]

Logout. Now I am a guest. Can see all the pictures and that is the way I think it should be. Can not see Nep's graph in the post above. Could it be that guests can see JPG images but not PNGs?

See for your self. Logout. Click here for a example.
https://endless-sphere.com/forums/viewtopic.php?f=1&t=14748&start=200#p1702660

Upload 10,000 records I stole from e-beach.

Wondering how many people are looking at this forum? Here:
https://www.phpbb.com/community/index.php
At the bottom I see:
Who is online

In total there are 1511 users online :: 28 registered, 1 hidden and 1482 guests (based on users active over the past 30 minutes)
Most users ever online was 27085 on Wed Aug 21, 2019 4:40 am

 

[marty]

Could it be that guests can see JPG images but not PNGs?

Wrong. Logout again. Can not see [10000 LP Records.JPG]
[You do not have the required permissions to view the files attached to this post.]

Do see [10000 LP Records.JPG] in the Re: Adding pictures? thread.

Something is not as it should be.