fechter said:
I think you'd need written permission from Ken to re-post the material.
The notice was another failed after thought by Ken, who btw unsubscribed from the V Refuge Monday, January 22, 2007 3:47 PM.
He was aware we had the backups, I told him how many and where they were, in case he needed them to repair the site. The only backup he found that was good at the time only went back to about June 2006. He also urged people to visit the temporary site until his site was repaired.
Currently we have about 700 MB of compressed data and images, and another 150 MB of compressed data we might need to re download from Germany. Together they should cover everything from Apr 2 2004, to the last day the site was up. The data before that is lost. Since all the pages are there, our only concern would be the few that have the statement on them.
There are now less than 30 of the original 60 days remaining, for the V is for Voltage Community to decide what they want to do with the domain and the backups, since they were the ones that initially requested both.
Posted: Apr 2 2004, 07:28 AM
Voltage Forum Admin
Group: Admin
Posts: 587
Member No.: 1
Joined: 2-April 04
We're back online! It's been a long week.
I guess we'll have to call this the Voltage Forum v1.1 or something now...
What happened to the site:
We experienced a very rare hardware failure that fatally corrupted not only the fault tolerant computer hard drive array, but also the onsite backup data. After failing to be able to restore the data in-house, We sent the drives out to data recovery specialists at considerable expense in order to try and salvage the site's datastructure and databases, but the data recovery guys were ultimately unable to recover any of our information. Because I had not fully debugged the system yet, I hadn't yet set up any offsite backups. So all our early data and all site infrastructure was completely lost. Our entire site needed to be recreated from absolute scratch. While the underlying server failure was out of my control, I am sincerely sorry that I did not take greater steps sooner to protect the first three weeks of site data. The lack of additional offsite backups was my failure as an administrator. It won't happen again.
Where we are now and what you need to do:
I have gotten the basic structure recreated and the forums are now back online. Not all functionality has been fully restored across the site, so things will continue to get refined for several weeks probably. All users will have to recreate their user accounts and can begin posting immediately. All moderators will need to email me directly with your moderation board preferences so that I can set you up with moderation status again in the appropriate places. All dealers will need to email me directly and I will put you into a special user group so that you can access the "dealers-only" forums. If anyone sees something that is missing or otherwise needs to be fixed, email me directly and I'll add it to the list.
admin@visforvoltage.com
How I prevent this from ever happening again:
This is what I have been working hard on for the past few days since the crash. First is a server upgrade. We are adding a second array controller and an additional array for some more fault tolerance within the server itself. We are also implementing a multi-layer backup strategy including:
a) improved onsite backup scripts and hardware in the data center in Toronto
b) nightly offsite backups to a server that I set up at a totally different ISP in Wisconsin
c) weekly hard copy backups onto CDROM at my local site in Washington
This multi-layered, geographically diverse approach affords us many site data snapshots, which in turn provide us with multiple fall back positions in case of catastrophic hardware failure like we saw in this case. This represents a significant expense in terms of both resources and bandwidth costs, but it is the only way I will be satisfied with the underlying safety of our forum data. All these measures are now in place and fully operational. We successfully ran a full backup test where we completely destroyed all site data including the databases and we then restored the entire site from a backup snapshot.
Why all this is expense and overkill is necessary:
Our site skyrocketed up over a million hits in the first three weeks of march with the Voltage Forums and V is for Voltage v3.0 site going online. We watched the traffic just build and build across the board with all the new and exciting community features. I am looking at this failure as a wake up call. This site is going to see some major growth and some SERIOUS traffic in the coming months. In the first three weeks alone, we served over 6GB of data with no video streams or large images. That's a large amount of bandwidth, and it's just gonna grow larger in the future. I had to make sure that our systems are ready for this kind of explosive growth.
Final thoughts:
I very much regret that this whole incident happened, but I honestly believe that we are now in a much safer position because of the improvements we've made in our systems in response to this crash. This kind of total data loss should not be able to happen again. As much as the members may feel cheated due to the loss of data, let me assure you that I am personally feeling it harder than anyone. Life goes on, you rebuild with a positive attitude, and you try to make sure it never happens again. That's what I am doing.
I welcome your thoughts and comments.
--------------------
- KT Roughneck
Outlaw Turbologist
206.339.8658 - vmail/FAX
ktrough - mobile AIM/YM
ken@visforvoltage.com
V is for Voltage Magazine
Hacker-Nitition Posted: Apr 3 2004, 01:15 PM
Voltage Forum Admin
Group: Admin
Posts: 587
Member No.: 1
Joined: 2-April 04
Regarding room for growth:
Our first three weeks netted over a million hits and about 6GB of total forum traffic.
We now have backup servers, a multiple offsite backup methodology, and a bandwidth overhead allowance of 60GB per month. I anticipate that this will serve our needs for at least 6 months, and if we need to beef up our back end some more, we have a scalable upgrade path at this point which can be implemented with almost no notice.
I think we are sitting really good at this point. I remain committed to the success and growth of this community. I also remain committed to keeping this forum free to end users. We certainly accept donations, but will never require $$ for access. The entire V is for Voltage site is dedicated first and foremost to being a global free library resource.
--------------------
- KT Roughneck
Outlaw Turbologist
206.339.8658 - vmail/FAX
ktrough - mobile AIM/YM
ken@visforvoltage.com
V is for Voltage Magazine