System downtime on 11/18/2020

This section will contain posting guidelines for the forum along with any other information that is relevant to this forum.
Post Reply
User avatar
neptronix   100 GW

100 GW
Posts: 14433
Joined: Jun 15 2010 5:56pm
Location: California refugee living in Utah, USA
Contact:

System downtime on 11/18/2020

Post by neptronix » Nov 18 2020 6:53pm

Well that was a white knuckle moment!
Amazon bricked the bootloader of our server after an update and i had to pull the disk and reconstruct the system.
Amazon's disk performance limits meant that this reconstruction took about 5 times longer than it should have while copying files.

I have to say i am not pleased with amazon right now and will not be updating the server or even rebooting it until we move away from this host.

Sorry for the interruption. I got us back up as fast as possible.
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500.
The new all-arounder: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: 20" eZee on a Cannondale Semi Recumbent.
Whipper-snapper: ? on a lightweight BikeE Semi Recumbent

User avatar
MJSfoto1956   100 kW

100 kW
Posts: 1066
Joined: Jul 28 2010 9:28pm
Location: Boston, MA

Re: System downtime on 11/18/2020

Post by MJSfoto1956 » Nov 18 2020 9:12pm

Oracle Cloud Infrastructure is likely much cheaper than Amazon -- especially for data storage.
Pretty much everything you need is there.
A little more "technical" than Amazon though.

M
"Will you shut (TF) up, man?"

QANON is a sick cult: https://www.rollingstone.com/culture/cu ... e-1064076/

User avatar
neptronix   100 GW

100 GW
Posts: 14433
Joined: Jun 15 2010 5:56pm
Location: California refugee living in Utah, USA
Contact:

Re: System downtime on 11/18/2020

Post by neptronix » Nov 21 2020 11:43am

Thanks for the tip. I didn't even know that Oracle did cloud hosting. It does look more like they're oriented towards larger organizations, but i like the modularity of their offerings. And pricing. Will keep that in mind. :)

I've had clients on AWS since 2015. It's been a super smooth ride with AWS until this year. I think they are not extensively testing their custom patches to mysql and the linux kernel. This year, i've had to rescue two machines from an unbootable state, and had a remote database that had redundancy go down for over an hour with no explanation. I had to pay for Amazon's tech support in order to ask why the database went down. The answer i received came very late and they blamed me for the outage, despite all evidence showing the problem was at their datacenter.

I think AWS has gotten cocky because they have a majority marketshare over web hosting lately.

Anyway between moving to another host soonish, upgrading to the latest php3.2.x branch, and also implementing some pretty badass anti-DDOS techniques i've been trialing on 2 other clients' servers, ES should see unprecedented levels of stability.

There are some bigger tricks up my sleeve for 2021 :thumb:
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500.
The new all-arounder: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: 20" eZee on a Cannondale Semi Recumbent.
Whipper-snapper: ? on a lightweight BikeE Semi Recumbent

Post Reply