• Howdy! we're looking for donations to finish custom knowledgebase software for this forum. Please see our Funding drive thread

Move to XenForo software

Status
Not open for further replies.

neptronix

Administrator
Staff member
Joined
Jun 15, 2010
Messages
18,282
Location
Utah, USA
Hey all.

An ongoing issue with phpbb's youtube embeds has lit a fire on my ass to start building a test site for ES 2.0 running XenForo instead of phpbb.

I am building the server today, will have the platform installed, and will start a test instance of our entire setup.. hopefully by EOD.
 
Test server up and running. :thumb:

server.png
 
Spent a number of hours getting a version of MySQL that works on both XenForo and phpBB.. fun times :lol:
 
Development copy of endless sphere finally working on new mysql.

Xenforo gettin' uploaded.

up.png
 
Why i hate AWS web hosting in one image:

slowdisk.png

Seeing write speeds average 12-15mb/sec on the most expensive and fast SSD they have.. with all the performance settings jacked up as far as they go. Occasionally i see peaks in the 100mb's for a second, and the CPU is averaging 50% load on both cores..

I've got a 7 gigabyte database to write to disk and this takes an hour or more.. i could have got this done faster on my linux desktop.

NOT the best hosting... but probably the most stable... WAY more stable and consistent than the digitalocean server we're on.. but damn, i want to run one of my own server farms one of these days.

Just waiting on the database crunching before i take a full disk backup and then hit the shiny button.

2021-02-05 16_54_11-XenForo 2.2.3 Patch 1 - Welcome _ XenForo.png
 
Got hit with the amazon disk speed throttle real bad.. 4 hours i'm still waiting on a nice database crunch..

Ah well, i'll be back on the task this weekend.
 
You're welcome.

Last night i encountered the usual massive MySQL write speed issues that cause all kinds of server operations to happen at a crawl.

I've taken the slow backup restore process down from 5 hours to 12 minutes after working on tuning MySQL all morning.
I've cut dozens of hours off the xenforo migrating, testing, etc process. Very 'stoked', as old surfers say on the central coast :mrgreen:

2021-02-06 13_59_49-Volumes _ EC2 Management Console.png

I'm truly shocked by the results. The high volume tune did not benefit my big server administration client nearly this much. It confirms to me that the database design of phpbb is far from optimal when operated at scale.

I wish i knew about this 5 years ago!!! unfortunately, information on the 'net about optimizing phpbb performance is lacking because most people run small forums that died off due to hacking, updates breaking things, etc.. so the collective knowledge never got to this point.
 
The project is a bit stalled as i take care of my clients today. But will continue in a few days.

Over the weekend, i did some work on my PHP framework, zerolith. It's a framework that's designed to plug into ANY system running on php, whether it has a framework or not. I am building some commercial software on it, and also using it to do things like log emails, debug messages, clean up databases, etc for many of my clients.

The process of migrating to XenForo, and the process of developing custom code that adds functionality to XenForo will benefit from this greatly.

This week, i created a debugger which will provide as a better aid than XenForo's built in debugger; since we're going to have LOTS of bugs up front, it'd be best to trace them down them as quickly as possible.

img.png

Anyway, more later.
 
Thanks a lot for this hard work, I can't wait until the forum is being updated, I'm sure it will be awesome!

One stupid question maybe: do you know if we will lose the pictures? that's generally a big issue I saw on some other forums, sometimes after migration the pictures links won't work anymore, which is a huge problem for build threads and tutorials for example.

Again, thanks a lot for the great work and for maintaining this forum!
 
Dui said:
One stupid question maybe: do you know if we will lose the pictures? that's generally a big issue I saw on some other forums, sometimes after migration the pictures links won't work anymore, which is a huge problem for build threads and tutorials for example.

Again, thanks a lot for the great work and for maintaining this forum!

That's not a stupid question. Preserving images is of utmost importance. We have lost some images along the way also due to database and file system issues that are common with phpbb whenever you jostle any setting.

Before we move, i have some large scale data processing tasks to partake in to try to turn external hosted images into local ones, and recover whatever file pointers have been lost to database corruption in the past.
 
nicobie said:
How about old PM's? will they transfer?

I have no idea yet. If they cannot be transferred, some way will have to be invented for you to list and read 'legacy PMs'.
We have hundreds of megabytes of PMs, so it's very important to preserve that.


As for the project status...

My client schedule this week isn't super packed, so there'll be another burst of activity this week on this. I'm also getting gammaray on task to work with me so we can tag team some things.

The goal for the end of the week is to have a working xenforo conversion of the forum, sans all the customizations. We'll come up with a list of what data transfers and what doesn't, and formulate a plan to handle that best.

If there is anyone here who would like to volunteer to help test these things, please raise your hand.
I would bet amberwolf would be willing, but i haven't asked him yet.
 
Sweet. Anyone else?

I'll come up on instructions for those volunteering for what to test, or you can simply free range if you prefer.
The more eyes on it, the better. And different eyes will see different things. :)
 
neptronix said:
I'll come up on instructions for those volunteering for what to test, or you can simply free range if you prefer.
The more eyes on it, the better. And different eyes will see different things. :)

Depends on the testing. I'm really not a programmer, but I could try out a beta site if that's what you mean.
 
Dauntless said:
Depends on the testing. I'm really not a programmer, but I could try out a beta site if that's what you mean.

That is what i mean, sorry for not being clear.

Basically i'll hand over a password to access ES 2.0 'beta' which will have a functional conversion of ES' content on it.
You'd need to poke around, and come up with a list of what's broken.

Once we have a list of things that are broken, it's my turn to fix 'em.
Then we do another round of poking around to confirm the fixes.
Then we do another round to confirm all the fun plugins we'll be installing don't have any goofy side effects.

Each round takes an hour or so, and you can do one round, two, three, whatever.
 
Applied some database optimizations to the current server to speed up dumping the DB and hopefully give us more stable/consistent performance on digitalocean while we're on here. Their slow IO has been plaguing our server since we moved there.

We had a little ~10 minute glitch while i did it.
 
Still getting a little disk i/o wait episodes but they're not as bad. OK, ES is stabilized. 8)
 
When we move over, perhaps borrow this quote to stick in the header:
“ Let us stand on each other’s shoulders, not each other’s toes”
Dennis Allison

Maybe it would help with the attitude problem a few of the people have around here (mostly those that end up in OTD a lot, but there are others....) :)
 
Also, when the memberlist is migrated, I would recommend not migrating any zero-post members. This will remove a huge amount of members from the list, most of whom were spammers.

Can't really not migrate banned members, unless they are separated from their posts, because there are quite a few banned members who left behind a lot of useful posts. ;)
 
If you don't migrate the banned members, aren't they then unbanned? Safe, Bluto, and more that people could fill in names for, unleashed all at once.

Who ya gonna call?

GHOST BUSTERS!
 
Probably an unusual topic for a first post, but as a new member that has referenced the forum extensively in the last few months, I thank you for your effort in updating it - and to everyone who provides the content/community. :)

A community I was part of a few years ago migrated from phpbb to Xenforo as well and I helped with bug testing. If there is anything I can do, let me know.

I'm not technically proficient in how Xenforo works on the back-end, but just relaying information I've observed based on inquires in this topic so far:
  • When the forums were being moved to Xenforo, the biggest effort/headache was getting all the uploaded files/images (there were 100'000s) to sync up with the topics they were posted in. For whatever reason all the topics 3 years or older didn't show the files/images that had been posted in them - this could just be due to the admin who wasn't experience with Xenforo.
  • The PMs from the old site were able to be migrated to the new site.
  • I think the banned users from the old forum had to manually have their settings altered on the Xenforo forum in order to be banned again. Iirc, it had something to do with the differences between how Xenforo and phpbb organize data differently. As that was a few years ago, maybe there is a better way to do that now.

Cheers :bolt:
 
Status
Not open for further replies.
Back
Top