Major database mess...I think?

mbrown

So, over the weekend I got reports that my disciples tools instances were, well, unusable. The sites did not render correctly. I did some digging and found that there were TONS of errors in the php error_log.

Looking at the logs it appears that some kind of a database schema migration is getting kicked off - probably part of some kind of theme update. It's definitely not wordpress core - I dont think wordpress core would involve things like this:

[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Starting migrating Network Dashboard to number 0
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Done migrating Network Dashboard to number 0
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Starting migrating Network Dashboard to number 1
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Done migrating Network Dashboard to number 1
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Starting migrating Network Dashboard to number 2
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Done migrating Network Dashboard to number 2
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Starting migrating Network Dashboard to number 3
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Done migrating Network Dashboard to number 3
[21-Aug-2024 14:57:54 UTC]  2024-08-21 14:57:54 GMT Starting migrating Network Dashboard to number 4

I'm also seeing a bunch of lines like this:

[21-Aug-2024 03:16:36 UTC] PHP Warning:  Undefined array key "group" in /home1/mjbhostc/public_html/equipleaders/dt/wp-content/plugins/disciple-tools-network-dashboard/snapshots/snapshot-queries.php on line 190
[21-Aug-2024 03:16:36 UTC] PHP Warning:  Undefined array key "group" in /home1/mjbhostc/public_html/equipleaders/dt/wp-content/plugins/disciple-tools-network-dashboard/snapshots/snapshot-queries.php on line 187

Most disturbingly:

[21-Aug-2024 14:57:55 UTC]  2024-08-21 14:57:55 GMT Starting migrating Network Dashboard to number 6
[21-Aug-2024 14:57:55 UTC] WordPress database error Unknown column 'al.field_type' in 'where clause' for query
           SELECT
                DISTINCT( al.object_id ) as site_object_id,
                'new_coaching' as action,
                al.hist_time as timestamp
            FROM TYE_8_dt_activity_log as al
            LEFT JOIN TYE_8_dt_movement_log as m
                ON al.object_id=m.site_object_id
                AND m.site_id = 'e88a1bbb3b80442deee7df00915de7247f16637d296f4204b5f0af7760477542'
                AND m.action = 'new_coaching'
            WHERE al.object_type = 'contacts'
                AND al.meta_key = 'contacts_to_contacts'
                AND al.field_type = 'connection from'
                AND m.id IS NULL
            ORDER BY al.object_id; made by require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('after_setup_theme'), WP_Hook->do_action, WP_Hook->apply_filters, {closure}, DT_Network_Dashboard_Migration_Engine::migrate, DT_Network_Dashboard_Migration_0006->up, DT_Network_Activity_Log::query_new_coaching
[21-Aug-2024 14:57:55 UTC]  2024-08-21 14:57:55 GMT Done migrating Network Dashboard to number 6

Now, I maintain a GFS rotation of backups for three months. So I went back and checked - every one of my backups has this stuff in the error logs. I previously had auto-updates enabled for both WordPress Core and plugins, and so I deduce that somehow something went very wrong in some of these updates, and I didn't know about it until things went even more wrong over this past weekend.

I restored back one by one until I found the most recent backup that still appeared to work.

Has anyone seen anything like this? This is a multisite instance too, and I'm dreading trying to clean it up.
Can anyone give me some guidance, should I decide to reinstall WP/DT and migrate the data to a new clean database, how I would do that?

Thanks in advance.

mbrown

@mbrown

I don't know if there was something stale in the cron triggers or what, but I did discover that Bluehost (my provider) had added a plugin that was forcing auto-updates and a few other things that I think were conflicting.

Well, I went back and restored backups until I found a working one again, made a bunch of tweaks to make sure the stupid bluehost plugin is disabled for good, and blocked all auto updating with filters in a mu-plugin.

So far so good. Started up with a bunch of errors related to a dt snapshot job of some sort but they haven't repeated.

Time will tell. Again, if anyone has seen anything like this, I'd appreciate the info.

Thanks!