Issue with some previous youtube embeds

This section will contain posting guidelines for the forum along with any other information that is relevant to this forum.
Post Reply
User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Issue with some previous youtube embeds

Post by neptronix » Feb 05 2021 12:19pm

Our forum is not showing youtube embeds from the past due to a database change from a plugin i attempted to install.
Current youtube embeds are safe to make and will work into the future.

I am working on a fix right now, which requires me to make large sweeping database changes.

Here is some corny hold music for you to listen to while i work on this.

"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Re: Ongoing issue with past youtube embeds - fixing

Post by neptronix » Feb 05 2021 12:45pm

Apparently some old youtube embeds work and others don't.

Clicking 'edit' and then 'save' on a post will fix the ones that are busted in the meantime.
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Re: Ongoing issue with past youtube embeds - fixing

Post by neptronix » Feb 05 2021 1:44pm

Okay, it looks like changing the youtube embed settings ( unexpected side-effect of installing advanced editor ) jostled phpbb into embedding things in a new way which is incompatible with future exports of our forum. The phpbb software wanted to do things this way but did not make the change through successive upgrades we've made.

phpbb has also historically stored youtube embed data in various different ways over the years.
That will have to be reconciled with some database-fu in the future.

I would restore parts of our system from backups, but many changes were made last night which i'd prefer not to revert and lose. The process would likely result in an even bigger mess than current.

A fire is lit under my ass to work on the new forum format starting today.
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

User avatar
amberwolf   100 GW

100 GW
Posts: 33740
Joined: Aug 17 2009 6:43am
Location: Phoenix, AZ, USA, Earth, Sol, Local Bubble, Orion Arm, Milky Way, Local Group
Contact:

Re: Issue with some previous youtube embeds

Post by amberwolf » Feb 06 2021 12:56am

FWIW, stuff like that is why I would never work on the "live" forum, but only on a copy, then when it was verified by at least several different people (with different workflows, viewpoints, methods, etc) to work without breaking stuff, *then* I'd apply the same stuff to the "live" forum.

I have no idea how practical this is...but even though it would be a bit more time-consuming initially, it would certainly save heartaches and headaches like this one. ;)
If you found this advice helpful, supporting contributions are accepted here.

Beginning of a New Life


Give a listen to my music!

(More here, including experimental)
.

User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Re: Issue with some previous youtube embeds

Post by neptronix » Feb 06 2021 3:40am

amberwolf wrote:
Feb 06 2021 12:56am
FWIW, stuff like that is why I would never work on the "live" forum, but only on a copy, then when it was verified by at least several different people (with different workflows, viewpoints, methods, etc) to work without breaking stuff, *then* I'd apply the same stuff to the "live" forum.
You gotta understand that making a copy of our database takes 5 hours per change you need to test for, alone. Then it requires a ~1 hour check of the site functionality at a minimum, maybe longer. I have found you can't cut corners on this time sucker by occasionally skipping the data copy process, either, due to the wide range of unexpected side effects in phpbb's code and data storage techniques.

Meanwhile, installing a plugin takes about 1-2 minutes. Pretty tempting, isn't it? Cleaning up a mess it makes takes anywhere from minutes to hours. Whereas testing takes a minimum of 5 hours, up to about 7. Testing also does not guarantee that you find all the problems. You may find them months into the future.

I'm aware this is not the ideal way to do this, but as you can see, the time economics are hugely slanted.

I have been trying to tackle the ES development problem since i took over in 2015. Of the perhaps 100 hours spent on working on ES, around 60 have been spent trying to make testing faster. I even went so far as to write something faster than Oracle's mysqldump in the process of trying to speed up database operations, which are the primary bottleneck to testing. That's how desperate i was.

I have a faster way to refresh a testing environment quickly with our next setup; it's something i discovered working with a client recently. So does Gammaray. Between these two ideas, i believe one will land. We haven't invested in this in our current setup as phpbb is not worth investing in, generally.


If you would like to volunteer to do testing, you could ensure this is done up to your standards. It is the most dull and un-gratifying job to most, so i've never asked anyone to help with it. How do you feel about doing some of this work?
amberwolf wrote:
Feb 06 2021 12:56am
I have no idea how practical this is...but even though it would be a bit more time-consuming initially, it would certainly save heartaches and headaches like this one. ;)
Actually the heartbreak to reward ratio on this one is well worth it because i will have to fix youtube embeds anyway. There's 5 different ways our system has done youtube embeds over time, and they all have to transformed before moving to a new platform. This changes my order of operation more than anything else.
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

User avatar
marty   100 MW

100 MW
Posts: 2596
Joined: Apr 19 2007 5:44pm
Location: Buffalo, New York USA
Contact:

Re: Issue with some previous youtube embeds

Post by marty » Feb 06 2021 8:04am

neptronix wrote:
Feb 05 2021 12:19pm
Here is some corny hold music for you to listen to while i work on this.
Try listening to this. Might help with computer confusion?


If that don't work? Try this.
Matteo's Mobile Bicycle repair
https://matteosbikerepair.com

User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Re: Issue with some previous youtube embeds

Post by neptronix » Feb 06 2021 2:04pm

Lol, thanks for the inspirational music.

I made a mistake in reading ES before i went to bed, and the comment from AW got under my skin, even though it wasn't meant as an attack. The frustration is more with the platform. I was up till 3am thinking of how to solve our issue with handling phpbb's data structure and the difficulties imposed by it.

There is a way to get faster disk access on amazon; you expand the size of the disk to many multiples of what you need. You back yourself into a corner of higher operating costs and the extra resources are only used 0.1% of the time. You make doing disk image backups exponentially slower, to the point of being useless. You make exporting a virtual machine for a testing environment impossible. The only way to revert this change is to have hours of downtime. It's awful. But you gain the ability to spawn a fresh testing environment in about 1.5-2 hours. When it becomes easier to test, it'll happen much more often.

I have initiated this change. It will take the entire day to benchmark the performance differences. If the extra resources become a budgetary stress, i will cover the additional cost while we are doing heavy construction. We can


The current plan for handling the youtube video issue is:

1) Transform old youtube tags into new tags universally.
2) Write some code which forces phpbb to display a random selection of posts that include
3) Write some code that translates these youtube tags back to the old style tags which xenforo can then import.
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

User avatar
neptronix   100 GW

100 GW
Posts: 16355
Joined: Jun 15 2010 5:56pm
Location: Utah, USA
Contact:

Re: Issue with some previous youtube embeds

Post by neptronix » Feb 06 2021 3:55pm

I have some VERY INTERESTING findings.

After installing a tune i use for another client with an wildly resource-intensive system, there was an immediate speed impact which cut the database copy time from 5 hours to 1. This increased speed took my testing time from 'all day' to hours. I was shocked.

Through progressive tuning, i have dropped this speed to 12 minutes. At the expense of high perpetual use of RAM; which is less expensive than all the extra disk i'd need.

I'm absolutely blown away at how much tuning mysql's innodb cache sizes, changing the write method to O_DIRECT, etc produces.

This REALLY big for ES. This enables me to spawn a fresh development system in about 20 minutes total. When i do that, performance of the forum is extremely poor, but stable. It also means that restoring from backup can be performed within minutes, not hours. Restoring from backup has always been a problem due to the amount of downtime it creates.

I can now craft specialized backup and restore procedures for specific aspects of the database, which would allow us to quickly recover from issues with the DB, such a plugin corrupting data/changing the storage technique.


These changes will translate to an enhanced development, maintenance, and user experience on XenForo as well.

That was absolutely worth losing sleep over :lol: :lol: :lol:
"Love and compassion are necessities, not luxuries. Without them, humanity cannot survive." - Dalai Lama

My first major build: 1.6kW 8T MAC motor on a Trek 4500 MTB.
Monster MTB: Leafmotor 1500w @ 4kW on a Turner O2 full suspension.
The monster scooter: Cannondale semi recumbent - under construction.
Blue Dream: Maxaraya FS semi recumbent and high efficiency mid-drive - under construction.

Post Reply