Move to XenForo software

[neptronix]

Hey all.

An ongoing issue with phpbb's youtube embeds has lit a fire on my ass to start building a test site for ES 2.0 running XenForo instead of phpbb.

I am building the server today, will have the platform installed, and will start a test instance of our entire setup.. hopefully by EOD.

 

[neptronix]

Test server up and running. :thumb:

 

[neptronix]

Spent a number of hours getting a version of MySQL that works on both XenForo and phpBB.. fun times :lol:

 

[neptronix]

Development copy of endless sphere finally working on new mysql.

Xenforo gettin' uploaded.

 

[neptronix]

Why i hate AWS web hosting in one image:



Seeing write speeds average 12-15mb/sec on the most expensive and fast SSD they have.. with all the performance settings jacked up as far as they go. Occasionally i see peaks in the 100mb's for a second, and the CPU is averaging 50% load on both cores..

I've got a 7 gigabyte database to write to disk and this takes an hour or more.. i could have got this done faster on my linux desktop.

NOT the best hosting... but probably the most stable... WAY more stable and consistent than the digitalocean server we're on.. but damn, i want to run one of my own server farms one of these days.

Just waiting on the database crunching before i take a full disk backup and then hit the shiny button.

 

[neptronix]

Got hit with the amazon disk speed throttle real bad.. 4 hours i'm still waiting on a nice database crunch..

Ah well, i'll be back on the task this weekend.

 

[nicobie]

Sounds irritating as hell. I'm sure all the members will be thankful for the new features. I know I will.

Much thanks for your time spent on this necessary upgrade.

 

[neptronix]

You're welcome.

Last night i encountered the usual massive MySQL write speed issues that cause all kinds of server operations to happen at a crawl.

I've taken the slow backup restore process down from 5 hours to 12 minutes after working on tuning MySQL all morning.
I've cut dozens of hours off the xenforo migrating, testing, etc process. Very 'stoked', as old surfers say on the central coast



I'm truly shocked by the results. The high volume tune did not benefit my big server administration client nearly this much. It confirms to me that the database design of phpbb is far from optimal when operated at scale.

I wish i knew about this 5 years ago!!! unfortunately, information on the 'net about optimizing phpbb performance is lacking because most people run small forums that died off due to hacking, updates breaking things, etc.. so the collective knowledge never got to this point.

 

[neptronix]

The project is a bit stalled as i take care of my clients today. But will continue in a few days.

Over the weekend, i did some work on my PHP framework, zerolith. It's a framework that's designed to plug into ANY system running on php, whether it has a framework or not. I am building some commercial software on it, and also using it to do things like log emails, debug messages, clean up databases, etc for many of my clients.

The process of migrating to XenForo, and the process of developing custom code that adds functionality to XenForo will benefit from this greatly.

This week, i created a debugger which will provide as a better aid than XenForo's built in debugger; since we're going to have LOTS of bugs up front, it'd be best to trace them down them as quickly as possible.



Anyway, more later.

 

[Dui ni shuo de dui]

Thanks a lot for this hard work, I can't wait until the forum is being updated, I'm sure it will be awesome!

One stupid question maybe: do you know if we will lose the pictures? that's generally a big issue I saw on some other forums, sometimes after migration the pictures links won't work anymore, which is a huge problem for build threads and tutorials for example.

Again, thanks a lot for the great work and for maintaining this forum!

 

[nicobie]

How about old PM's? will they transfer?

The reason I ask is that I have a few old ones that have links in them I would like to keep. If I know that they will go away in the transfer, I will take the time to look them up and store them somewhere else.

 

[neptronix]

One stupid question maybe: do you know if we will lose the pictures? that's generally a big issue I saw on some other forums, sometimes after migration the pictures links won't work anymore, which is a huge problem for build threads and tutorials for example.

Again, thanks a lot for the great work and for maintaining this forum!

That's not a stupid question. Preserving images is of utmost importance. We have lost some images along the way also due to database and file system issues that are common with phpbb whenever you jostle any setting.

Before we move, i have some large scale data processing tasks to partake in to try to turn external hosted images into local ones, and recover whatever file pointers have been lost to database corruption in the past.

 

[neptronix]

How about old PM's? will they transfer?

I have no idea yet. If they cannot be transferred, some way will have to be invented for you to list and read 'legacy PMs'.
We have hundreds of megabytes of PMs, so it's very important to preserve that.


As for the project status...

My client schedule this week isn't super packed, so there'll be another burst of activity this week on this. I'm also getting gammaray on task to work with me so we can tag team some things.

The goal for the end of the week is to have a working xenforo conversion of the forum, sans all the customizations. We'll come up with a list of what data transfers and what doesn't, and formulate a plan to handle that best.

If there is anyone here who would like to volunteer to help test these things, please raise your hand.
I would bet amberwolf would be willing, but i haven't asked him yet.

 

[nicobie]

No worries! It will get done sooner or later.

I'd be happy to help out any way I can.

Nick

 

[neptronix]

Sweet. Anyone else?

I'll come up on instructions for those volunteering for what to test, or you can simply free range if you prefer.
The more eyes on it, the better. And different eyes will see different things.

 

[Dauntless]

I'll come up on instructions for those volunteering for what to test, or you can simply free range if you prefer.
The more eyes on it, the better. And different eyes will see different things.

Depends on the testing. I'm really not a programmer, but I could try out a beta site if that's what you mean.

 

[neptronix]

Depends on the testing. I'm really not a programmer, but I could try out a beta site if that's what you mean.

That is what i mean, sorry for not being clear.

Basically i'll hand over a password to access ES 2.0 'beta' which will have a functional conversion of ES' content on it.
You'd need to poke around, and come up with a list of what's broken.

Once we have a list of things that are broken, it's my turn to fix 'em.
Then we do another round of poking around to confirm the fixes.
Then we do another round to confirm all the fun plugins we'll be installing don't have any goofy side effects.

Each round takes an hour or so, and you can do one round, two, three, whatever.

 

[neptronix]

Applied some database optimizations to the current server to speed up dumping the DB and hopefully give us more stable/consistent performance on digitalocean while we're on here. Their slow IO has been plaguing our server since we moved there.

We had a little ~10 minute glitch while i did it.

 

[neptronix]

Still getting a little disk i/o wait episodes but they're not as bad. OK, ES is stabilized. 8)

 

[kauaicycler]

I'm willing to beta test.

 

[neptronix]

I'm willing to beta test.

Cool, thank you. :thumb:

 

[amberwolf]

When we move over, perhaps borrow this quote to stick in the header:
“ Let us stand on each other’s shoulders, not each other’s toes”
Dennis Allison

Maybe it would help with the attitude problem a few of the people have around here (mostly those that end up in OTD a lot, but there are others....)

 

[amberwolf]

Also, when the memberlist is migrated, I would recommend not migrating any zero-post members. This will remove a huge amount of members from the list, most of whom were spammers.

Can't really not migrate banned members, unless they are separated from their posts, because there are quite a few banned members who left behind a lot of useful posts.

 

[Dauntless]

If you don't migrate the banned members, aren't they then unbanned? Safe, Bluto, and more that people could fill in names for, unleashed all at once.

Who ya gonna call?

GHOST BUSTERS!

 

[Jaraldo]

Probably an unusual topic for a first post, but as a new member that has referenced the forum extensively in the last few months, I thank you for your effort in updating it - and to everyone who provides the content/community.

A community I was part of a few years ago migrated from phpbb to Xenforo as well and I helped with bug testing. If there is anything I can do, let me know.

I'm not technically proficient in how Xenforo works on the back-end, but just relaying information I've observed based on inquires in this topic so far:

• When the forums were being moved to Xenforo, the biggest effort/headache was getting all the uploaded files/images (there were 100'000s) to sync up with the topics they were posted in. For whatever reason all the topics 3 years or older didn't show the files/images that had been posted in them - this could just be due to the admin who wasn't experience with Xenforo.

• The PMs from the old site were able to be migrated to the new site.

• I think the banned users from the old forum had to manually have their settings altered on the Xenforo forum in order to be banned again. Iirc, it had something to do with the differences between how Xenforo and phpbb organize data differently. As that was a few years ago, maybe there is a better way to do that now.

Cheers