Forum broke again

[marty]

It started Saturday 8/24/24 at about 8:30 pm. Spins in circles for a while then error page. [The connection has timed out] or something like that? Copied the text and deleted by accident. Problem continued for all of Sunday 8/25/24 till last night when endless-sphere.com reappeared. So it was down for about 24 hours.

I am using Mozilla Firefox. Was able to connect to every other web site on the internet.

Thanks for keeping things running.

 

[marty]

Not my error page. Image I found on the internet. This is what I was seeing.

 

[neptronix]

Anyone else notice this?
We had healthy traffic all weekend and i received no alerts of our system being down.

Two possibilities:
1) Some ISP interconnections had issues.
2) You triggered our DDOS protection mechanism somehow.

Were you using a mobile Apple device btw?

 

[Mongo]

Anyone else notice this?
We had healthy traffic all weekend and i received no alerts of our system being down.

Two possibilities:
1) Some ISP interconnections had issues.
2) You triggered our DDOS protection mechanism somehow.

Were you using a mobile Apple device btw?

I didn't notice and connection issues or timeouts, and I was hopping on & off the forum throughout the weekend.

 

[marty]

Were you using a mobile Apple device btw?

No. Just so you know. There are two types of people. Android and Apple. I am neither. I have a flip phone. View internet on big house computer.

 

[marty]

If no one else was having connection issues? I would guess that a wire between the endless-sphere.com server.

Did some spying on you. See Amazon.com, Inc. in a few different cities around Oregon. Portland, Salem, and Boardman.

I am over at the other end of the US. Verizon FIOS comes into my house from the pole behind house. Guessing something went wrong with the wires between the Amazon server, Verizon, and my house. My guess? Squirrels chewed on a wire. Or humans got stupid with a back hoe. See map of where wires are.

 

[neptronix]

Yeah. I see this happen every once and a while. usually it's a problem with the internet tubes in the user's locale.
I call it the .05% part of the 99.95% uptime guarantee AWS provides.

 

[marty]

10/1/2024 6:30am. Forum moving real slow. Been getting this notice. Page could not be loaded. Could be something to do with rabbits and turtles?

 

[marty]

7:00am Forum now moving fast like normal. My guess is too many people or robots were looking at endless-sphere.com

 

[marty]

7:04am It's slow again. Problem is intermittent.

 

[harrisonpatm]

Same for me as well, when I was checking at 4AM and 6AM Eastern, but seems to be back to normal now

 

[LewTwo]

Anyone else notice this?
We had healthy traffic all weekend and i received no alerts of our system being down.

Affirmative ... I think about 3:00 AM US Central time. I got the same error screen Marty posted. No problem elsewhere (i.e. Yourube, Amazon). I am running the Google Chrome browser on Linux Mint OS. ISP is Verizon via a Orbit3 hot-spot.

 

[neptronix]

Morning. I'm awake and at the controls.
Thanks for the report. I did get a ton of alerts in the morn too about slow performance.

It appears that xenforo doesn't have a retention policy for it's mail logs, and they reached 128gb in size in the 1.5 years we've ran this system.

Apparently a common problem for some small percentage of Xenforo users with high traffic.

The backup process collided with another scheduled backup process and both were running at the same time, which murdered the CPU.

I'm truncating that database table, the forum may freeze up periodically because it's a large operation. Will update when done.

 

[neptronix]

Done

 

[neptronix]

Designed a retention policy for that data and some others. Will implement that later. Should be a permafix for this problem.

 

[harrisonpatm]

Thanks!

 

[Papa]

ISP is Verizon

https://downdetector.com/status/verizon/

 

[neptronix]

https://downdetector.com/status/verizon/

The problem was at the server level.

 

[LewTwo]

https://downdetector.com/status/verizon/

Yes ... but my disdain of the local cable companies (Spectrum in Mission, Tx) and their crappy installation practices knows no bounds. When I moved into this place, I pulled out over 300 feet of various coax cables that they had run. Much of it on top of the ground. My lot is only 36 x 50 feet, so that is a lot of useless coax.

 

[chuyskywalker]

Designed a retention policy for that data and some others

Sysadmin 101, been there, done that.

"Systems' funky." "Logs?" Proceed to find 500GB log file filing up all possible drive space

 

[neptronix]

That's the funny thing, we had enough disk space, it's just that one database table crossed the threshold of mysql being able to process it!

But yes that is the bane of my existance and i'm determined to finish a disk space monitoring system by the end of the year.

 

[chuyskywalker]

that one database table

Oh, yeah -- or when they are missing that one critical index