Update on Outage, Lost Images, Parse errors, etc.

[justin_le]

Hi everyone, so I am finally back in town and unpacked and just somewhat settling in again. There have been plenty of fires to deal with on several fronts and unfortunately the transfer of ES to a new server is chief among them. In a nutshell it turned out that due to the combination of bad circumstances and what may be called a typo in a file saving command, the attachment database with all the images and files that went live with ES was an incomplete set (only had about 25K of the 50K files), and the full backup of all the attachments was saved in a way that makes it very difficult to extract and re-link all of the files. This was realized by MrVass just moments after the original server had erased and re-imaged all their data. He's lost a lot of sleep over the situation, and has been working hard along with the help of some other forum members to set things straight again.

The current status is that we have the full (but somewhat corrupted) backup set from early in August, and we also have all of the uncorrupted images from the electric bicycles forum that the user Vodka has most kindly had saved and made access too. So in the near future we expect to have all of the older images and attachments which are missing go live again along with a fix for most of the corrupted images.

HOWEVER, there is a 2-3 week time window at the end of August where there may be files and images which we don't have at all, and the only way that these will get restored is if people re-upload them. This is going to be a drag I am sure and I apologize in advance, but since they are all quite recent posts I am hoping that most affected people still have their attachment files conveniently handy. We'll know exactly within the next couple days.

The recent outage around 2pm PST on Sept 12th was due to us moving around large backup files to/from the server while working on these issues and temporarily exceeding a quota, and the error message that shows up when a post is submitted is something that we're aware of too and will get fixed.

This has been a much rockier start than I would have liked and I feel like we've let the community down quite a bit, and I take full responsibility for that. As MrVass mentioned our own expertize and skill set is very much in hardware design and embedded systems, not web applications, and I had a lot of hesitancy about getting involved with ES because of this. But in the end thought there was too much to loose by not getting involved.

So it has certainly been a steep learning curve to get fully up to speed, though we're committed to getting there.

-Justin

[Jozzer]

Thank you Justin, we appreciate your efforts and trust you will all do your best to fix it all up as best as possible. There were bound to be some problems with such a big move!

Steve

[TylerDurden]

Thanks for the update!

So IIUC, images (and attachments?) prior to August will be restored? i.e. We need not manually replace those?

If so, the attachments that are gone forever might be easily identified by mod/users if the attachment icon could be displayed per-post when a member's posts are searched. Currently, the attachment icon only is displayed at the category level.

e.g. I can search my own posts by clicking on my username in the memberlist... "search all member's posts" shows a list of every post I have made, but does not identify which posts have attachments. If I could see an attachment icon (paperclip) on the posts with attachments, I could quickly check if the attachments need replacing.


This phpbb mod may do the trick, but I'm not running a board to test it.

http://phpbb.davidiq.com/viewtopic.php?t=142

Maybe another member running phpbb can try?


Thanks!



edited for better clarity.

[Toorbough ULL-Zeveigh]

there's no need to feel like you've disappointed anyone.
if it's simply a matter of time before the bulk of the attachments are restored i think that's great & have no problem waiting.
if all it comes down to is losing only a couple of weeks worth of pictures then that's a cheap price to pay compared to what we might have lost.
well, cheap for the rest of us anyways, for you probably not so much.

[Miles]

e.g. I can search my own posts by clicking on my username in the memberlist... "search all member's posts" shows a list of every post I have made, but does not identify which posts have attachments. If I could see an attachment icon (paperclip) on the posts with attachments, I could quickly check if the attachments need replacing.

If you go to User Control Panel | Manage Attachments (click twice on the Post Time heading), you can check the attachments directly. You also get direct links to all of the posts that they were attached to.

[Arlo1]

Thanks Justin.
No worries we are all in this together!

[RallySTX]

I'm very happy with your efforts, no issues here, GOOD JOB!
Brian L.

[Pure]

Thanks for all the hard work Justin. I agree, a couple of weeks worth of loss is pretty cheap compared to the alternative.

I never upload any images directly to any forum. I always use a photo hosting site like photobucket, and strongly urge others to do the same. It's not that I don't trust this or any other forum with my content. It's because it requires less resources from the forum, and in the unfortunate event that something like this does happen, and as long as my post wasn't altered, my images should never be lost. I also just find it a lot easier, as most sites have pic size limits that always seem to be smaller than my default settings. It's very tedious and time consuming if I have to always adjust my image size before uploading. Photobucket doesn't care how large my pics are, and I can then just direct link those pics in my post.

[TylerDurden]

e.g. I can search my own posts by clicking on my username in the memberlist... "search all member's posts" shows a list of every post I have made, but does not identify which posts have attachments. If I could see an attachment icon (paperclip) on the posts with attachments, I could quickly check if the attachments need replacing.

If you go to User Control Panel | Manage Attachments (click twice on the Post Time heading), you can check the attachments directly. You also get direct links to all of the posts that they were attached to.

Excellent.

Thanks Miles!

[dogman]

I hate to say it, but I figured there would be some learning curve involved with the new forum admin. But happily, you have quite the rep for learning fast, and making improvements.

No complaints from me. Now if only my new internet provider was the same way.

[liveforphysics]

Just knowing the forum is in good hands is all I need to be happy, even if those hands are still slightly forum-incompetent still. (not that i could do any better of course, just ribbing you a bit my friend.

Nothing but love for you guys, and whatever happens, happens. We know the place we have is safe, even if we lose an attachment or two once in a while.

[Arlo1]

Thanks for all the hard work Justin. I agree, a couple of weeks worth of loss is pretty cheap compared to the alternative.

I never upload any images directly to any forum. I always use a photo hosting site like photobucket, and strongly urge others to do the same. It's not that I don't trust this or any other forum with my content. It's because it requires less resources from the forum, and in the unfortunate event that something like this does happen, and as long as my post wasn't altered, my images should never be lost. I also just find it a lot easier, as most sites have pic size limits that always seem to be smaller than my default settings. It's very tedious and time consuming if I have to always adjust my image size before uploading. Photobucket doesn't care how large my pics are, and I can then just direct link those pics in my post.

I hate photo bucket you will knotice how many prople never click a link to a picture! And i belive photo bucket is what caused all Jeremy Harris's problems! Sorry Jerremy if i spelt your name wrong!

[rebelpilot]

I hate photo bucket

+1
Leaves a gaping hole if the OP moves on to other interests, and doesn't keep the photo account.

[Hillhater]

I hate photo bucket you will knotice how many prople never click a link to a picture!

what do you mean by .." how many people never click a link to a picture"... ?
With PB ( or Image Shack, etc ..plenty of options ) ,.. the image appears in the post !
Its the forum hosted images that seem to have the "link" ..sometimes !

If the forum software was really as smart as their creators think they are, they would automatically capture any images posted (from any source) and host a copy on their own server.

[methods]

Photobucket and the like are the worst thing that ever happened to forums. Just go browse threads from 5 years back and you will see threads riddled with holes from missing attachments. It may seem like a good idea now - but it does not persist over time. I either serve my own pictures or upload them to the forum directly.

As far as the hiccups with the forum?
phfffttt... this is nothing








-methods

[Pure]

Photobucket and the like are the worst thing that ever happened to forums. Just go browse threads from 5 years back and you will see threads riddled with holes from missing attachments. It may seem like a good idea now - but it does not persist over time. I either serve my own pictures or upload them to the forum directly.

As far as the hiccups with the forum?
phfffttt... this is nothing








-methods

Those problems are from people who move/delete their images after having linked them. I have images linked from over 4 years ago and to this day appear as they should. I must be an anomaly, you see once I link an image, I never delete it or move it. To this day I get emails asking me questions on thing's I've posted on forums I haven't visited in years. Now that I really think about it, every forum I have ever been on has lost data from both sources corrupted/lost attachments from server issues or PB noobs moving their images. So one is as bad as the other in the long run. For me it's a non issue. PB is easier to deal with when wanting to show images on a forum and I know at least "MY" images will never disappear.







That only took 30 seconds and will never disappear. The best part is I'm not using any of ES server resources on image storage. It really boils down to personal choice, and for the reasons I've listed I personally will never upload any of my images to any forum.

[Hillhater]

Also, using a separate image store keeps control of those images with the original poster.
So if some other jerk tries to take over a forum, you can pull down the images easily.

[biohazardman]

I would have to agree with pure on this one. I have many pics still active on other forums, hosted on photobucket, that are from the mid to early 1990s. I make a conscience choice not to move, re-arrange or modify them in any way so they stay functioning. I have no problems with them at all but then again I also take care of my account by logging in to it on a regular basis so it remains active. Photobucket has been great with the exception of all the advertisements there when I visit but they have to pay for it somehow and that's a small price to pay for what you get.

[dogman]

My only problem was that I stupidly did not back up my own files of photos. I thought, hey why bother they'll always be there on ES if my computer crashes. My computer did die, and then I never went and got the pix back. Then es lost em in a migration.

Happily, yesterdays problems seem to be gone. Yay GRIN.

[mrvass]

My only problem was that I stupidly did not back up my own files of photos. I thought, hey why bother they'll always be there on ES if my computer crashes. My computer did die, and then I never went and got the pix back. Then es lost em in a migration.

Happily, yesterdays problems seem to be gone. Yay GRIN.

Yes, you'll be happy to know that thanks to help from some awesome people, I'm now in the process of restoring the rest of the attachments.

That said, I highly recommend a personal offsite backup solution like rsync, idrive, or dropbox for your own files, as this allows more personal control over their fate. There are plenty of others out there too, I just haven't used them yet.

[GCinDC]

I'm VERY happy about the fixed images!

No pressure, but what's next? Should there be a thread to discuss continuing updates to the forum? FWIW, here are some ideas that I've had...

SOME ES FORUM IDEAS:
- vimeo embed tags? Be nice to be able to embed vimeo vids, like second vid here....
- RSS feeds from forum, as discussed here.
- Email alerts to topic updates - works pretty well, but could/should it include text of new post? (edit: searching phpBB, i don't see that capability but I do see adding user name of replier.
- Search: Ability to (advanced) search on a few characters, such as: "DH" or "FS" and include dashes in searches, though this might prove a performancerisk...
- Support for accents in user names: I've been seeing quite a few funny characters in user names; assume they're accents?
- Privacy settings? Guest access? Robots Enabled or Disabled? I don't have any preferences but might be good discussion topics...


ENDLESS-SPHERE APP ON ANDROID MARKET:
I'm not sure how the endless sphere app came out on the android market, but it's a nice way to read the forum, though it still needs a few updates for automatically refreshing, posting images, etc. But I assume it's using some other forum reader app.

WIKI:
Timeframe for getting that platform up?

[rodgah]

ENDLESS-SPHERE APP

tapatalk script for iphone users please!!! would make it awsome for me because 50% of my time is spent on a phone/tablet as I am away from home alot.

Cheers

[flyinmonkie]

Thanks for restoring the pictures. Awesome job, it is nice to have them back. I am excited to see where it all goes from here.

Clay

[Kingfish]

MRVass, do you know when the corrupted characters will be fixed? Do you need help with that?

Best, KF

[dogman]

A great many of my long lost pics are back. Some not, but many are there.

Dogman VERY HAPPY!

Quite a few are not important, but others are. Just found a pic of my worst ever build, the 5304 fuji wheeliemobile. That's the sort of think I was bummed to have lost, the pics of all the differnt weird stuff I tried in the last 4 years.

Pic of my first ebike.