Move to XenForo software

Status
Not open for further replies.
amberwolf said:
When we move over, perhaps borrow this quote to stick in the header:
“ Let us stand on each other’s shoulders, not each other’s toes”
Dennis Allison

Maybe it would help with the attitude problem a few of the people have around here (mostly those that end up in OTD a lot, but there are others....) :)

Something along those lines is a priority in the later stages of development. We should brainstorm some ways to nudge the tone of the forum towards more positive and engineering based.
 
Jaraldo said:
Probably an unusual topic for a first post, but as a new member that has referenced the forum extensively in the last few months, I thank you for your effort in updating it - and to everyone who provides the content/community. :)

A community I was part of a few years ago migrated from phpbb to Xenforo as well and I helped with bug testing. If there is anything I can do, let me know.

I'm not technically proficient in how Xenforo works on the back-end, but just relaying information I've observed based on inquires in this topic so far:
  • When the forums were being moved to Xenforo, the biggest effort/headache was getting all the uploaded files/images (there were 100'000s) to sync up with the topics they were posted in. For whatever reason all the topics 3 years or older didn't show the files/images that had been posted in them - this could just be due to the admin who wasn't experience with Xenforo.
  • The PMs from the old site were able to be migrated to the new site.
  • I think the banned users from the old forum had to manually have their settings altered on the Xenforo forum in order to be banned again. Iirc, it had something to do with the differences between how Xenforo and phpbb organize data differently. As that was a few years ago, maybe there is a better way to do that now.

First off, i really appreciate this being your first post. Welcome to the forum. :)
Thanks for the insight on how these conversions go. I'm glad the PMs come along.

Like AW mentioned, i can see why migrating with a clean user base would be ideal. I've already cleaned up the moderator team, so banned/inactive accounts should be purged as well.

I'd love to have you involved in testing. Thanks for raising your hand on that.

To everyone else, i have a chunk of time later in the week to restart the project. You'll hear from me later.
 
neptronix said:
Something along those lines is a priority in the later stages of development. We should brainstorm some ways to nudge the tone of the forum towards more positive and engineering based.
Here's some good "rules" we could emulate; they all seem reasonable to me, seem to cover most of the things we need to deal with, and better-worded than our own present ones (though we might need some of those carried over, too, particularly regarding spam, sales, etc):

https://forum.v1engineering.com/guidelines

This is a Civilized Place for Public Discussion

Please treat this discussion forum with the same respect you would a public park. We, too, are a shared community resource — a place to share skills, knowledge and interests through ongoing conversation.

These are not hard and fast rules, merely guidelines to aid the human judgment of our community and keep this a clean and well-lighted place for civilized public discourse.

Improve the Discussion

Help us make this a great place for discussion by always working to improve the discussion in some way, however small. If you are not sure your post adds to the conversation, think over what you want to say and try again later.

The topics discussed here matter to us, and we want you to act as if they matter to you, too. Be respectful of the topics and the people discussing them, even if you disagree with some of what is being said.

One way to improve the discussion is by discovering ones that are already happening. Spend time browsing the topics here before replying or starting your own, and you’ll have a better chance of meeting others who share your interests.

Be Agreeable, Even When You Disagree

You may wish to respond to something by disagreeing with it. That’s fine. But remember to criticize ideas, not people. Please avoid:

Name-calling
Ad hominem attacks
Responding to a post’s tone instead of its actual content
Knee-jerk contradiction

Instead, provide reasoned counter-arguments that improve the conversation.

Your Participation Counts

The conversations we have here set the tone for every new arrival. Help us influence the future of this community by choosing to engage in discussions that make this forum an interesting place to be — and avoiding those that do not.

Discourse provides tools that enable the community to collectively identify the best (and worst) contributions: bookmarks, likes, flags, replies, edits, and so forth. Use these tools to improve your own experience, and everyone else’s, too.

Let’s leave our community better than we found it.

If You See a Problem, Flag It

Moderators have special authority; they are responsible for this forum. But so are you. With your help, moderators can be community facilitators, not just janitors or police.

When you see bad behavior, don’t reply. It encourages the bad behavior by acknowledging it, consumes your energy, and wastes everyone’s time. Just flag it. If enough flags accrue, action will be taken, either automatically or by moderator intervention.

In order to maintain our community, moderators reserve the right to remove any content and any user account for any reason at any time. Moderators do not preview new posts; the moderators and site operators take no responsibility for any content posted by the community.

Always Be Civil

Nothing sabotages a healthy conversation like rudeness:

Be civil. Don’t post anything that a reasonable person would consider offensive, abusive, or hate speech.
Keep it clean. Don’t post anything obscene or sexually explicit.
Respect each other. Don’t harass or grief anyone, impersonate people, or expose their private information.
Respect our forum. Don’t post spam or otherwise vandalize the forum.

These are not concrete terms with precise definitions — avoid even the appearance of any of these things. If you’re unsure, ask yourself how you would feel if your post was featured on the front page of the New York Times.

This is a public forum, and search engines index these discussions. Keep the language, links, and images safe for family and friends.

Keep It Tidy

Make the effort to put things in the right place, so that we can spend more time discussing and less cleaning up. So:

Don’t start a topic in the wrong category.
Don’t cross-post the same thing in multiple topics.
Don’t post no-content replies.
Don’t divert a topic by changing it midstream.
Don’t sign your posts — every post has your profile information attached to it.

Rather than posting “+1” or “Agreed”, use the Like button. Rather than taking an existing topic in a radically different direction, use Reply as a Linked Topic.

Post Only Your Own Stuff

You may not post anything digital that belongs to someone else without permission. You may not post descriptions of, links to, or methods for stealing someone’s intellectual property (software, video, audio, images), or for breaking any other law.

Powered by You

This site is operated by your friendly local staff and you, the community. If you have any further questions about how things should work here, open a new topic in the site feedback category and let’s discuss! If there’s a critical or urgent issue that can’t be handled by a meta topic or flag, contact us via the staff page.

Terms of Service

Yes, legalese is boring, but we must protect ourselves – and by extension, you and your data – against unfriendly folks. We have a Terms of Service describing your (and our) behavior and rights related to content, privacy, and laws. To use this service, you must agree to abide by our TOS.
 
Wordy but good.. and not far off from where we already are. :thumb:

Changing rules and such is more of a forum design topic. This thread was meant to be about technical details; more of a progress log.
Let's have another discussion about that on a later thread, because that discussion is important to have.
 
Just an update for now.

I'm writing v3 of my image resizer. It is something i haven't wanted to do for years because testing it has been such a bother. But we're going to have to move ES to a new host, and also deal with the flood of image uploads we get in our new software. So the time to produce something that manages image sizes on a recurring basis is not only necessary now, but later.

This will later be released as an open source system that can adapt to whatever system you have that allows users to upload images. it uses advanced compressors to optimize file sizes.
 
Just blew 6 hours writing version 3 of the Image Tender.. damn, she's 400 lines of code already.

..it can now run on a recurring basis, test the input and output images for validity, operate on 100,000's of images in small chunks to prevent overloading the server, and can also be pointed at any app we use in the future ( wordpress, xenforo ) and perpetually resize and recrunch images using mozjpeg, gifsicle, and pngquant for optimum

..it will allow me to speed up the move to the new system and also ensure much lower hosting and backup costs for us.

It's about 88% done, just needs debugging and testing on 100,000's of images, but testing it is no longer a hurdle due to this new design.

2021-02-23 19_53_55-Courtesy VM [Running] - Oracle VM VirtualBox _ 1.png

I'm so curious how much it can cut down the ~50gb of images we have. Betting we'll see it cut our file storage in half.

Okay.. more later.
 
Jesus, i'm about 500 lines of code and 10 hours of work into this beast.
It needs just a little more debugging.

Along the way, i discovered at least 2 of ES' images are corrupt, in a sample of 100 images.
If i process these corrupt images, there's a risk i will destroy content. So my checks need to be good. I spent a lot of time writing jpeg, gif, and png checking.

There is a chance i can repurpose this batch image processor to fix images in the phpbb database, while i am at it.
So.. i probably have another day of writing code that does auditing and repair of the database before we get moving on testing the xenforo transition.

Also, look at how goofy phpbb's file storage format is. No extensions... so i have to read the header of the files to find out. :roll:

2021-02-24 21_07_29-Courtesy VM [Running] - Oracle VM VirtualBox _ 1.png
 
Version 3.2 is completed and it's a 700 line whopper.

Took my server 20 minutes to validate all the files in our storage... but here's the final report from my batch processor.

2021-02-25 16_44_18-Courtesy VM [Running] - Oracle VM VirtualBox _ 1.png

So, 2.85% of our images are corrupt. Which is roughly what i expected.
I thought perhaps the database markers to these files were missing, but no, it's file corruption.
Why this happened is a mystery. I have been in PhpBB's upload code before and seen some weird hacked together shit though. It doesn't validate uploaded images very well.

I'm going to add a little code to this batch processor to note the dates of corrupted files and place them in another folder so i can see if they can be recovered.

If anyone knows some jpeg recovery tools, i'd like to hear about them.
 
Ah yeah.. close to the money. I read the wiki article and it seems like more of a disk recovery type software. We're so many server moves away from the original server, that this can't help :/
Thanks anyway; that's an interesting tool to know about for the future if shit hits the fan with a disk.

Perhaps GD is a little too sensitive to faults in image files and something like irfanview could batch process them, since it seems to handle corrupted jpegs fairly well at the least.

Currently working on adding a flag in my batch processor to rename the extensionless files from 37112_78c742416ebd9a92707d9429c15fffe9 to 37112_78c742416ebd9a92707d9429c15fffe9.jpg, then copy them to a folder so that i can download them and see what the deal is. Some windows software would be OK to use then.
 
I don't know if it's useful, but I've used XnConvert in the past to process extensionless images. It's been long enough that I don't remember any details though, so it might not do anything useful for you.
 
amberwolf said:
I don't know if it's useful, but I've used XnConvert in the past to process extensionless images. It's been long enough that I don't remember any details though, so it might not do anything useful for you.

Oh yeah, some years ago i looked for a tool like that. Nothing could handle the combination of corrupt images, zip/pdf/other types, plus extensionless names *without* also dying from a memory leak in the middle of a job while processing our ~40gb of images, and not letting me know where it left off so i could continue after ( avoiding recompressing images and losing image quality in the process.. ) 250,000 images is a lot for most programs to handle unless they're specifically designed to handle stratospheric batch sizes.

( i had the same problem finding a text editor that could handle the 7gb dump from our database, lol. )

I actually developed two versions of my batch convertor for ES over the years. Each one had memory management or speed control issues and would have created big performance problems on the server while running. They also couldn't handle our corrupt images properly without barfing. This version is pretty gentle, has a tiny memory footprint, and really, really good throttling.

I also wanted something you can point at a folder or database of images and it'll perpetually manage files in that folder without recompressing them and losing quality of the image over successive passes. This means it has to track files using it's own database, or xenforo/phpbb's database. That requires lots of code, but... worth it for automating a task that takes a long time and shouldn't be subject to human error.
 
Okay, the batch processor throttling and database-oriented control is functioning properly. i have a folder of the unreadable images i need to look through and run through a 'fixer'.

A large portion of these corrupted images are thumbnails that phpbb generated. My guess is that phpbb does not validate images before and after processing them. The good news is that with some database work, these thumbnails can be tossed and forum posts can have the original, hopefully working image assigned instead.

All that's left of debugging tasks is to make sure the image resizing and recoding output looks OK visually.
I've worked on this for 3 straight days and need a break.

2021-02-26 12_17_09-Courtesy VM [Running] - Oracle VM VirtualBox _ 1.png
 
Well, we needed all of this 5 years ago.. it's been weighing on me this whole time to still not have a good platform to switch to.. and now i have the resources and the green light to do it. So getting the monkey off my back is pretty important to me.

The mini-sprints will continue until forum morale improves. :bolt:
 
pps - xenforo is badass and you are going to frocking love it. I can't wait to get it rolling. I have never felt so motivated to do anything in my life. Right now i'm forcing myself to ear the yogurt on top so i can savor the fruit on the bottom. Seriously.
 
Hoping to put in another round of work in on thursday/friday.


Another thing i will be developing is a tool for re-categorizing threads to aid in the creation of new subforms that are more specific than 'ebike technical' and 'ebike general'

This is easier to do with an 'aftermarket' tool in phpbb than xenforo, so it should get done ahead of the move.

Currently recruiting 1-3 volunteers for this task:
https://endless-sphere.com/forums/viewforum.php?f=50
 
I ran 8 guitar forums for 20 years. We started with Matt’s BBS, switched to phpbb, then vbulletin 3X for a dozen or more years and finally XF.

I know phpbb has improved over the years but XF is vastly better and customizable.

Be prepared for 10% to 15% to hate it and as many as 5% actually leave. Just hang tough and all will work out.

I hired a really great conversion tech and it was really a big help. Though 8 forums got expensive!

Good luck.

PS. Sold everything and retired in 2018.
 
Thank you so much for providing your valuable experience!

Moving to XF is an absolute and i am super positive about it.
I know there will be a degree of revolt when we switch over. It will also create a change in demographics. I'm very concerned about these things, but know it will be positive in the end.

Would you be willing to put me in touch with your forum conversion tech?
 
neptronix, I may be available on a limited basis for testing/feedback, my day job is full stack business reporting in tsql, vb, c#, crystal reports, SSRS, QlikView, etc.

I have no experience with any forum software directly, but I would be happy to to start with basic forum testing if you an instance running.

Cheers,

Adam

P.S. I really enjoyed reading "Cured my wife's anxiety and my depression with nutritional supplements" I started taking D3, it cleared up some skin problems, thx.
 
You sound like a great man for the job. Thank you for volunteering some of your time.

So far, we have for testing:
nicobie
dauntless
kauaicycler
Jaraldo
A-DamW
...and amberwolf, right? :)

Anyway, the project is a bit stalled at the moment due to a lack of time on my part. The image processor just needs testing a pinch of debugging and maybe some transcoding too. Super close to complete.

I'm making another call for programming help, which will be compensated for monetarily on a side gig basis, in the help wanted forum.
 
Just a quick state of the union..

I have been unable to find a chunk of time to work on ES, but my schedule is beginning to look more open next week. I would love to finish the image tender next week.

I have an assistant programmer, MrVass, from grin coming on to assist with some data processing tasks; thanks Justin.
I am also setting up an internship with another programmer.
I have a full staff for doing subforum re-arrangement and a full staff of testers.

I am impressed with how many people are willing to put some skin in the game for making this forum awesome.
For the first time i've came on, we have quite a good team to assist with technical matters.

There are way more considerations and cleanup before the xenforo move than i thought there would be. I think i should stop promising timelines. Just know that the ball is rolling pretty quick now. :)
 
Been a while since i gave an update. My work schedule is still totally nuts, but i'm starting to see things freed up.

I have MrVass and A-DamW both set up with development environments for ES data processing and they are starting to get going. I've decided to hand some xenforo migration tasks to them too. So they can help with the non-drudgework too. Very happy to have the help when my time is very limited. :)
 
Just a quick update.
A-DamW has done a test migration of phpbb to xenforo and the results of it's importation process are better than expected.

I'm particularly impressed with how refined XenForo is on the backend. For users, the search, private message, and posting system improvements are going to be a big deal :)
I'm compiling a full list of issues we need to tackle, and would like to get other testers involved when we are close to what we'd consider a 'beta'.

I'm firing one of my largest clients today and am overjoyed that i'll have more time to work on things like this.
I can't wait to give you guys this treat. :mrgreen:
 
Status
Not open for further replies.
Back
Top