Well that was a white knuckle moment!
Amazon bricked the bootloader of our server after an update and i had to pull the disk and reconstruct the system.
Amazon's disk performance limits meant that this reconstruction took about 5 times longer than it should have while copying files.
I have to say i am not pleased with amazon right now and will not be updating the server or even rebooting it until we move away from this host.
Sorry for the interruption. I got us back up as fast as possible.
System downtime on 11/18/2020
System downtime on 11/18/2020
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama
My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.
My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.
-
MJSfoto1956 10 kW
- Posts: 960
- Joined: Jul 28 2010 9:28pm
- Location: Boston, MA
Re: System downtime on 11/18/2020
Oracle Cloud Infrastructure is likely much cheaper than Amazon -- especially for data storage.
Pretty much everything you need is there.
A little more "technical" than Amazon though.
M
Pretty much everything you need is there.
A little more "technical" than Amazon though.
M
Bois and their fancy tacticool ain't cool -- it's a pathetic circle jerk
QANON is a sick cult: https://www.rollingstone.com/culture/cu ... e-1064076/
QANON is a sick cult: https://www.rollingstone.com/culture/cu ... e-1064076/
Re: System downtime on 11/18/2020
Thanks for the tip. I didn't even know that Oracle did cloud hosting. It does look more like they're oriented towards larger organizations, but i like the modularity of their offerings. And pricing. Will keep that in mind. 
I've had clients on AWS since 2015. It's been a super smooth ride with AWS until this year. I think they are not extensively testing their custom patches to mysql and the linux kernel. This year, i've had to rescue two machines from an unbootable state, and had a remote database that had redundancy go down for over an hour with no explanation. I had to pay for Amazon's tech support in order to ask why the database went down. The answer i received came very late and they blamed me for the outage, despite all evidence showing the problem was at their datacenter.
I think AWS has gotten cocky because they have a majority marketshare over web hosting lately.
Anyway between moving to another host soonish, upgrading to the latest php3.2.x branch, and also implementing some pretty badass anti-DDOS techniques i've been trialing on 2 other clients' servers, ES should see unprecedented levels of stability.
There are some bigger tricks up my sleeve for 2021
I've had clients on AWS since 2015. It's been a super smooth ride with AWS until this year. I think they are not extensively testing their custom patches to mysql and the linux kernel. This year, i've had to rescue two machines from an unbootable state, and had a remote database that had redundancy go down for over an hour with no explanation. I had to pay for Amazon's tech support in order to ask why the database went down. The answer i received came very late and they blamed me for the outage, despite all evidence showing the problem was at their datacenter.
I think AWS has gotten cocky because they have a majority marketshare over web hosting lately.
Anyway between moving to another host soonish, upgrading to the latest php3.2.x branch, and also implementing some pretty badass anti-DDOS techniques i've been trialing on 2 other clients' servers, ES should see unprecedented levels of stability.
There are some bigger tricks up my sleeve for 2021
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama
My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.
My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.