Progress 
Move to XenForo software
- Thread starter neptronix
- Start date
- Status
ZeroEm
1 MW
Should not keep neptronix in the server room for so long. I'm sure he needs a ride to relax.
markz
100 TW
Are the servers physically located at Grintech HQ, or is it offsite or outsourced to cloud host service/host company with remote online access?
I dont pay attention when the forum goes haywire to error screen.
Its all Greek to me.
I dont pay attention when the forum goes haywire to error screen.
Its all Greek to me.
marty
1 MW
Fire up your internet. In your favorite search engine type in ( find website host ) or something like that. I used:markz said:
https://sitechecker.pro/hosting-checker/
Hosting Checker
Find out who is hosting this website and where is it hosted
Type endless-sphere.com into the box. Copy and paste is faster.
Answer to your question?
Web Hosting Provider: Amazon.com, Inc.
Marty's prediction for the future. Marty wants to ship a box from here to there. I compare prices at usps.com, fedex.com, and ups.com. Amazon will start shipping packages like USPS, FedEx, and UPS.
Have you heard that FedEx, and UPS will be merging? The new name will be Fed Up. :lol:
markz said:
It's on amazon and unfortunately the phpbb 'stutter' is worse than it was on digitalocean :x
I have recently found out the source of this stutter and it is a design issue with phpbb.
Not worth fixing when the grass is much greener on the other side.
ZeroEm
1 MW
I know he is working remotely, just commenting on how hard he has been working. Notice he does not post much these days.
markz
100 TW
Yeah that's good to know how to check up on the ownership and who hosts a website.
marty said:
Oh yeah my business went exponential this year and i had to have a come to jesus moment with my stress levels and let two clients go a few months ago. I have been dealing with the backlog ever since and am regaining my footing and also taking first steps towards hiring my first employee.
In Utah, we've had horrible pollution this year; my brain has been hit/miss for months, and i also had the first athsma attack of my life while outside in the pollution, so i've stayed out of the warehouse space i'm paying for, as well as off a bike. No bike action for me until fall.
All work and no play makes jack a dull boy, but i am looking forward to fall
In Utah, we've had horrible pollution this year; my brain has been hit/miss for months, and i also had the first athsma attack of my life while outside in the pollution, so i've stayed out of the warehouse space i'm paying for, as well as off a bike. No bike action for me until fall.
All work and no play makes jack a dull boy, but i am looking forward to fall
marty
1 MW
Owner is who owns the domain name endless-sphere.com. I used to be able to look this up. Now it looks like the world has changed. Could be because of email spammers harvesting email addresses? Can't see domain name owners any more. According to my research the domain name endless-sphere.com comes from namecheap.commarkz said:
See:
https://www.namecheap.com/domains/whois/result?domain=endless-sphere.com
Results of the 10,000 image sample are in for the image processor.
30% reduction of our images storage needs means we can be even more liberal about image uploads in the future and get away with it. 8)
1.4 hours to process 10,000 images extrapolates to about 29.4 hours for all the images, which is a lot rosier than my projections. But that also means this will take a very long time to test.
I will start a test of the full image set today and manually review the results next week to look for any signs of image degradation. I will visually sample at least 200 images before/after.
I have some ideas also on how to improve on the 30% reduction rate by using metadata tag stripping tools in cases where recompressing an image leads to higher file sizes, and the system decides to keep the original. In this case, it will try the metadata stripping tool to achieve a smaller size. If that fails, it will keep the original file and mark it as processed.
Speaking of metadata, we will be stripping that from uploads to this forum so that when you upload a file, nobody can download it and find your location if you have a camera that tags your location to the photo. Some cameras attach data like this to their jpegs, and that data is retained if you upload straight from your camera without using intermediary software.
So we preserve your privacy and we save a sliver on hosting costs over time. Everyone wins.
30% reduction of our images storage needs means we can be even more liberal about image uploads in the future and get away with it. 8)
1.4 hours to process 10,000 images extrapolates to about 29.4 hours for all the images, which is a lot rosier than my projections. But that also means this will take a very long time to test.
I will start a test of the full image set today and manually review the results next week to look for any signs of image degradation. I will visually sample at least 200 images before/after.
I have some ideas also on how to improve on the 30% reduction rate by using metadata tag stripping tools in cases where recompressing an image leads to higher file sizes, and the system decides to keep the original. In this case, it will try the metadata stripping tool to achieve a smaller size. If that fails, it will keep the original file and mark it as processed.
Speaking of metadata, we will be stripping that from uploads to this forum so that when you upload a file, nobody can download it and find your location if you have a camera that tags your location to the photo. Some cameras attach data like this to their jpegs, and that data is retained if you upload straight from your camera without using intermediary software.
So we preserve your privacy and we save a sliver on hosting costs over time. Everyone wins.
neptronix said:
Think you're stressed now? :lol: :lol: :lol:
Go for it David! I think you'll make a good boss.
Thank you very much. Other people told me that.
This week has been too busy to hire yet again
:lol:
Anyway i have kicked the last small bugs out of the image processor after running into every possible corrupt file based snag..
..everything was cool until i was 95% of the way through the tests and encountered an animated gif in the images folder that makes the server's memory explode and the process die when attempting to parse it.. other command line recompressors had the same problem.. but this image falsely passes GD's sniff test. The image itself may be either corrupt or something malicious.
.. but that's 1 image out of ~215,000.
..anyway, i'm running a third 48hr pass through all the images so i can do the visual comparison of at least 200 images, before and after. Hoping i can move on to the easier tasks soon.
:lol:
ComradeM is working on the youtube tag fixes and has been hitting some snags, but i have a good feeling he can pull it off.
This week has been too busy to hire yet again
:lol: Anyway i have kicked the last small bugs out of the image processor after running into every possible corrupt file based snag..
..everything was cool until i was 95% of the way through the tests and encountered an animated gif in the images folder that makes the server's memory explode and the process die when attempting to parse it.. other command line recompressors had the same problem.. but this image falsely passes GD's sniff test. The image itself may be either corrupt or something malicious.
.. but that's 1 image out of ~215,000.
..anyway, i'm running a third 48hr pass through all the images so i can do the visual comparison of at least 200 images, before and after. Hoping i can move on to the easier tasks soon.
:lol: ComradeM is working on the youtube tag fixes and has been hitting some snags, but i have a good feeling he can pull it off.
So we are getting close?
Yes, in terms of how many man-hours are required, i'd say we're 70% or more done with the prerequisites.
The Youtube tag fix is 1/10th as hard as this but requires a lot of testing time and maybe even a unique way to automatically test it ( we could perhaps hijack phpbb's post parser code as a means to test the output )
The ban import should require maybe a dozen lines of code tops and is an easy slam dunk.
After these things are done, we have to do a rehearsal xenforo move and transformation to prove the process before we put the live system through it.
The Youtube tag fix is 1/10th as hard as this but requires a lot of testing time and maybe even a unique way to automatically test it ( we could perhaps hijack phpbb's post parser code as a means to test the output )
The ban import should require maybe a dozen lines of code tops and is an easy slam dunk.
After these things are done, we have to do a rehearsal xenforo move and transformation to prove the process before we put the live system through it.
Good news indeed!
Much appreciation to all involved. The effort will be well worth it.
Much appreciation to all involved. The effort will be well worth it.
TheBeastie
1 MW
Is Amazon AWS really that good value? I been out of the hosting/web business for a while now so I don't know what is a good deal.neptronix said:markz said:
It's on amazon and unfortunately the phpbb 'stutter' is worse than it was on digitalocean :x
I have recently found out the source of this stutter and it is a design issue with phpbb.
Not worth fixing when the grass is much greener on the other side.
But seems like just everyone is on Amazon these days and nothing else, so I assume it must be a great deal...
I used to be senior admin for a publicly listed company that ran some popular services (about 10 years ago now) that did about one million page loads per day (not impressions but actual jsp/servlet page loads), when I left the company a new executive insisted on AWS and after moving some small projects to Amazon the new executive wasn't pleased when there was a budget blowout on the AWS.
But after I left I never knew where it's all gone due to moving into different things.
TheBeastie said:
All depends on your use case and how you set it up. AWS has tons of variables to account for in the pricing.
Uptime is great. I've only had an Amazon-based problem 3 times over 6 years, and i manage about 16 servers these days.
Generally speaking i would rather start a smaller client off with vultr if they don't have complex hosting needs like we do. Easy to get good performance per $ there.
Our system is on AWS for one reason: i can buy a slow magentic hard disk to store very very large backups for a long period of time at a reasonable cost; and those backups are also easy/fast to access.
Another good thing about AWS is that you can generally scale up and down with the click of a button. Can't do that on many other hosts.
AWS wins on flexibility, is pretty good in value, but i generally steer my clients clear of them because i don't want them to turn into a monopoly on web hosting.
Digitalocean is my least favorite host due to the unreliability of it's disk speeds.
TheBeastie said:
You can absolutely do AWS wrong in terms of pricing if you don't know how to turn it's knobs right. I've seen people get into that situation many times. It did actually take me a while to learn how to optimize costs on AWS.. they don't make it very easy to find such things out.
TheBeastie
1 MW
Yeah I wondered if that had changed at all, since I last used it.neptronix said:
I didn't have access to cost knobs or anything, I believe me and devs were given a sub-account from exec above me where we could just deploy a new 3 options of VM and we couldn't see anything else.
One of our devs deployed a bunch of new VMs for some java projects, I believe he deployed the biggest VM of the 3 options.
About a month later our new exec expressed frustration at the AWS cost blowout.
I just thought it was kind of crazy that we couldn't even see or have any idea of what kind of cost we were racking up, and I assumed that things would of changed.
It's remarkable at the thought AWS hasn't changed at all in terms of snake-costing and is still such a successful business, I would of assumed the cloud competitors would of crushed them with more transparent costing ability.
When I moved the email system to rackspace.com for the MS exchange style experience with Office Outlook office users what I liked about RS web UI control panel was that is every single new email account or increase in email storage space option was clearly labeled with the price increase it would cost. It was basically impossible to have a surprise cost blowout.
I assumed when looking at that RS system control panel for email/costs that cloud services would copy it so there would be no surprises.
What AWS really gives you is an extreme amount of flexibility and more features than you know what to do with. The degree of granularity in all the settings is what creates huge variability in your costs. But this is also it's advantage.
I agree about rackspce. I used to really like them.. they had the best interface i've seen. AWS has a clunky and buggy interface.
I think rackspace lost on a price basis..
AWS can give great value if you use reserved instances and commit to a >1 year term.
I agree about rackspce. I used to really like them.. they had the best interface i've seen. AWS has a clunky and buggy interface.
I think rackspace lost on a price basis..
AWS can give great value if you use reserved instances and commit to a >1 year term.
Holy shit. The results are in.
Before: 53gb
After: 26gb
Other than an additional temp file in the processed folder, the file count is the same, and there is no difference in the number of 0 byte files either.
Results are particularly dramatic on large uploads:
No perceptible image degredation in jpegs using mozjpeg versus phpbb's GD:
And one big UGH. When i strip jpeg metadata, the meta tag that says "rotate the image this way because i'm too lazy to do it while writing the file", typical of mobile phone generated jpegs, the processor ends up "rotating" some images.
Results on .png files that should have been uploaded as jpegs are dramatic.. about a fourth of the size. No perceptible quality loss here. What can i say, pngquant is a beast and genius piece of code.
In the inverse situation, where a jpeg should have been a png or gif, mozjpeg handles it quite well and the image is almost a tenth the size:
Given that i see an overall 50% reduction in file size, i think i can afford keeping the jpeg metadata to prevent photos from being "rotated" :lol:
I could probably bring the 50% down to 33% or less using brute force transcoding and PSNR checks afterward, but this processor already takes a minimum of 29 hours to run so.. I think this is a reasonable technique.
Before: 53gb
After: 26gb
Other than an additional temp file in the processed folder, the file count is the same, and there is no difference in the number of 0 byte files either.
Results are particularly dramatic on large uploads:
No perceptible image degredation in jpegs using mozjpeg versus phpbb's GD:
And one big UGH. When i strip jpeg metadata, the meta tag that says "rotate the image this way because i'm too lazy to do it while writing the file", typical of mobile phone generated jpegs, the processor ends up "rotating" some images.
Results on .png files that should have been uploaded as jpegs are dramatic.. about a fourth of the size. No perceptible quality loss here. What can i say, pngquant is a beast and genius piece of code.
In the inverse situation, where a jpeg should have been a png or gif, mozjpeg handles it quite well and the image is almost a tenth the size:
Given that i see an overall 50% reduction in file size, i think i can afford keeping the jpeg metadata to prevent photos from being "rotated" :lol:
I could probably bring the 50% down to 33% or less using brute force transcoding and PSNR checks afterward, but this processor already takes a minimum of 29 hours to run so.. I think this is a reasonable technique.
By the way, what this means from a user standpoint is that you can upload 20mb 9000 x 9000 jpegs all day and never have to worry about us being able to handle image storage.
I took the image upload limits off phpbb last year because i knew this system would get completed eventually.
Those limits will also not exist on Xenforo.
My thinking is that a forum needs to be as easy and convenient as using say, facebook.. the user shouldn't have to think about details like that.
I took the image upload limits off phpbb last year because i knew this system would get completed eventually.
Those limits will also not exist on Xenforo.
My thinking is that a forum needs to be as easy and convenient as using say, facebook.. the user shouldn't have to think about details like that.
That's a significant difference. Well done!
- Status