| @traylenator:matrix.org | > <@clarkb:matrix.org> given that we think this may be a bit less of a crisis than perviously anticipated I guess I'll context switch to cleaning up Steve Traylen's old gerrit account. I've found the ID for the old account in Gerrit's error_log and found my notes from the last time I had to do this. | 00:56 |
|---|---|---|
| Clark: I am able to login in now . Thank you for sorting this for me. | ||
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/zone-opendev.org] 985619: Add zp03.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/985619 | 05:43 | |
| -@gerrit:opendev.org- Michal Nasiadka proposed: [opendev/system-config] 985620: Add zp03.opendev.org https://review.opendev.org/c/opendev/system-config/+/985620 | 05:44 | |
| @gthiemonge:matrix.org | hi, i was stuck for +/- a minute on the Anubis page while trying to reach gitea (with "Calculating... Difficulty: 4, Speed: 0kH/s"), I reloaded 2 or 3 times then it passed, is it expected or a bug? | 12:58 |
| @gthiemonge:matrix.org | or am i a bot? | 13:01 |
| @fungicide:matrix.org | gthiemonge: it should normally take less than a second, so not expected at least. also it should bypass anubis in that browser for subsequent requests so it would only come up once | 13:14 |
| @fungicide:matrix.org | it looks like load may have spiked up massively on the gitea11 backend and then haproxy took it out of the pool (15-minute load average is 66, 5-minute load average is 0.0) | 13:17 |
| @fungicide:matrix.org | so if gthiemonge was being persisted to gitea11 while it was overloaded before it got removed, that could account for the observed symptom i think | 13:18 |
| @clarkb:matrix.org | the calculation occurs client side too so it could be completing then the redirect is what was failing but not in a way that makes that clear? | 13:56 |
| @fungicide:matrix.org | yes, that's basically in line with my suspicion as to how it happened | 13:58 |
| @jim:acmegating.com | I'm restarting the zuul schedulers/web to pick up a bugfix that recently landed | 13:59 |
| @mordred:waterwanders.com | fungi: I think you discounted the possibility that gthiemonge is a bot too quickly | 14:07 |
| @fungicide:matrix.org | us bots gotta stick together | 14:18 |
| @jim:acmegating.com | #status log restarted zuul schedulers/web to pick up a recent change cache bugfix | 14:19 |
| @status:opendev.org | @jim:acmegating.com: finished logging | 14:19 |
| @clarkb:matrix.org | I noticed this when I upgraded Gerrit too, but Zuul scheduler and web restarts are pretty quick these days | 14:19 |
| @clarkb:matrix.org | Zuul just had a 14.1.0 tag pushed so I checked Gerrit caches again to see if anything changed with its git-tags h2 v2 cache. Doesn't appear to have done so. The timestamp on that file is still from when mnasiadka restarted Gerrit yesterday and `gerrit show-caches` continues to show it with 0.00k usage. That said the tag itself replicated and I don't see any complaints in the error_log that seem related to that. So I'm going to consider this working even if it isn't clear why | 15:41 |
| @fungicide:matrix.org | the old infra-root mailbox and subfolder contents are finally all cleared out now, with the relevant messages moved to similar paths in the mailbox at the new provider, as previously agreed | 15:43 |
| @mnasiadka:matrix.org | Clark: I was looking at mediawiki patches - but I noticed that the Ansible role patch is failing on the missing image - is there any logic in some other job that just builds the image locally when it's missing in quay.io - or you always rely on the image in quay.io? | 15:51 |
| @clarkb:matrix.org | mnasiadka: the images will be pulled from the insecure-ci-registry if they exist there. That gets pruned though and had pruning issues | 15:52 |
| @fungicide:matrix.org | so may need to recheck earlier changes in series to refresh it, i guess? | 15:53 |
| @clarkb:matrix.org | mnasiadka: I think if you ensure the change that builds the image has run recently then the child changes should be able to find that image in insecure-ci-registry. If they cannot then it amy mean that provides/requires are not set up properly | 15:53 |
| @clarkb:matrix.org | fungi: yup I think so | 15:53 |
| @clarkb:matrix.org | we had issues with pruning over the holidays that caused things to get pruned too aggressively and that has been fixed so newer builds should be more reliable. And if the image can't be found after a rebuild then we need to look at provides and requires | 15:53 |
| @mnasiadka:matrix.org | Clark: In theory there's soft dep in here: https://review.opendev.org/c/opendev/system-config/+/921322/38/zuul.d/project.yaml but it has not run (see https://review.opendev.org/c/opendev/system-config/+/921322/38?tab=change-view-tab-header-zuul-results-summary) - maybe it works only if you introduce the job in the same patch? | 16:00 |
| @clarkb:matrix.org | mnasiadka: those dependencies are for job order. For the artifact tie in we need to use provides requires | 16:00 |
| @clarkb:matrix.org | mnasiadka: the image build job provides the job artifact and the job that uses the image requires the artifact | 16:01 |
| @mnasiadka:matrix.org | Ok, from what I see theres provides: in the images patch (https://review.opendev.org/c/opendev/system-config/+/921321/19/zuul.d/docker-images/mediawiki.yaml) and requires: in the Ansible role patch (https://review.opendev.org/c/opendev/system-config/+/921322/38/zuul.d/system-config-run.yaml) | 16:03 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 16:04 | |
| @mnasiadka:matrix.org | (fixed the 1.35 to require the 1.35 image) | 16:04 |
| @mnasiadka:matrix.org | * (fixed the 1.35 to require the 1.35 image - but that was only a typo) | 16:04 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 16:06 | |
| @clarkb:matrix.org | maybe see if ^ has any different behavior now that it looks like images were built yesterday. It looks like the failure ran roughly concurrently so maybe there is a bit of a chicken and egg and with the recent rebuild it may be happier? I don't know, but we should have more info shortly | 16:10 |
| @mnasiadka:matrix.org | Let's see, it might be that. | 16:13 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 16:32 | |
| @mnasiadka:matrix.org | hmm, docker-compose.yaml was referencing docker.io/opendevorg/mediawiki, maybe that's the issue | 16:33 |
| @clarkb:matrix.org | oh yes that would do it | 16:33 |
| @clarkb:matrix.org | fungi: ok it is ~1700 which is when we thought we might start no the trove db upgrade bandaid ripping process. Do you think you want to use the process described in the email they sent? Or do you think we want to create a new 5.7 db and do more of a traditional stop service, dump db, load db, start service against new db process? | 17:01 |
| @clarkb:matrix.org | and do you want to start with cacti or wiki? I'm happy to let you drive this as I think you've done far more investigative work than I have at this point. But do let me know how I can be helpful | 17:02 |
| @clarkb:matrix.org | (and maybe you need to eat lunch first?) | 17:03 |
| @fungicide:matrix.org | i need to grab a shower finally now that i have a break, but i think there's not a lot of option to roll back at the trove end so in the case of the wiki it would look more like dumping into the local mysql and reconfiguring to use it instead | 17:05 |
| @fungicide:matrix.org | the answer for the cacti server is a little less rosy since the mysql version on it is already basically the same as what we'd upgrade trove to | 17:06 |
| @clarkb:matrix.org | Ok that makes sense for wiki. For cacti I think we do the same approach for consistency. Then nothing is in trove any longer? | 17:06 |
| @clarkb:matrix.org | But cacti is also half config managed so may need to cross check that | 17:06 |
| @fungicide:matrix.org | since cacti is not user-facing and we're begrudgingly okay with a gap in data there, we have some breathing room. for wiki it's been widely announced as best-effort for years so a bit of unannounced downtime should also be okay | 17:07 |
| @clarkb:matrix.org | I can look at the config management for cacti and see if it is reasonable to do from that perspective | 17:07 |
| @clarkb:matrix.org | I will probably take a break too | 17:07 |
| @fungicide:matrix.org | i think the option for the least effort is to take final backups and then upgrade trove and see if things still work, then if they don't we can work through the local fallback option | 17:08 |
| @fungicide:matrix.org | but i'll mull it over in the shower | 17:08 |
| @clarkb:matrix.org | Ack. I'll look at cacti to see what if anything is done with the database server to see how feasible the local option is there | 17:09 |
| @fungicide:matrix.org | basically if making bigger changes is unnecessary then i'd prefer not to waste our valuable time overengineering for one service that's slated for replacement and another service that has a complete revamp in flight | 17:10 |
| @fungicide:matrix.org | for mediawiki it seems like it should "just work" anyway with 5.7, and for cacti you mentioned a possible workaround if it breaks on the backward incompatibility were worried about | 17:12 |
| @clarkb:matrix.org | Yup I posted a forum link in here with a server SQL mode that apparently makes 5.7 work | 17:13 |
| @clarkb:matrix.org | https://forums.cacti.net/viewtopic.php?t=11262 this post | 17:32 |
| @clarkb:matrix.org | fungi: ok I've dug through system-config/modules/openstack_project and I don't see us managing mysql at all for cacti locally and I don't see is managing /etc/cacti/debian.php which is where we set the cacti db connection details. So I think we can pretty safely (without ansible or puppet intervention) edit that configfile to point to a new db location and migrate to the local database if we want to. But I agree starting with the trove system as that seems maybe easiest is probably a good starting point and take it from there | 17:39 |
| @clarkb:matrix.org | also cacti is a long term resident of the emergency file already. So ya I think we can proceed and figure out how to fix things as they break | 17:41 |
| @fungicide:matrix.org | okay, so here's a lightweight wiki database upgrade plan for ~immediate execution... | 17:52 |
| @fungicide:matrix.org | 1. status notice that it's going down for a few minutes | 17:52 |
| @fungicide:matrix.org | 2. stop apache, both to prevent further edits and free up resources on the server/db | 17:52 |
| @fungicide:matrix.org | 3. both snapshot the trove instance in rackspace and mysqldump to the server, in parallel | 17:53 |
| @fungicide:matrix.org | 4. upgrade the trove instance | 17:53 |
| @fungicide:matrix.org | 5. reboot the server (for hygiene, we've seen some weirdness with session management in the past that reboots seem to clear, i never got to the bottom of it but just want to avoid it if possible) | 17:54 |
| @fungicide:matrix.org | 6. browse around, make a test edit | 17:54 |
| @clarkb:matrix.org | I think you may need a 5.5 based on the email rax sent and that is updating the connection string for the application? It looks like the upgrade process will give you a new server? But that isn't 100% clear to me. Something to check as we go through it | 17:55 |
| @fungicide:matrix.org | 7. if it's not working, send another status notice, stop apache again and work on loading the db backup into the local mysqld, update config, et cetera | 17:55 |
| @fungicide:matrix.org | yeah, i don't think it has done so in the past, but i'll double-check that | 17:55 |
| @clarkb:matrix.org | that plan looks reasonable to me | 17:55 |
| @clarkb:matrix.org | then once that is done I guess we rinse and repeat with cacti (though that doesn't need status notices) | 17:56 |
| @fungicide:matrix.org | yeah, cacti process would be way more cut-down | 17:57 |
| @fungicide:matrix.org | i'll go ahead and get logged into rackspace and check back over the announcement from them one last time | 17:58 |
| @fungicide:matrix.org | #status notice The wiki.openstack.org service will be offline for a few minutes while we perform a database upgrade | 18:05 |
| @status:opendev.org | @fungicide:matrix.org: sending notice | 18:05 |
| -@status:opendev.org- NOTICE: The wiki.openstack.org service will be offline for a few minutes while we perform a database upgrade | 18:09 | |
| @status:opendev.org | @fungicide:matrix.org: finished sending notice | 18:09 |
| @fungicide:matrix.org | stopping apache now | 18:09 |
| @fungicide:matrix.org | doing database backups | 18:10 |
| @clarkb:matrix.org | I'm following along. Let me know if I can be helpful | 18:10 |
| @fungicide:matrix.org | backup snapshot in rackspace is called `Wiki-MySQL-backup_2026-04-21_before_upgrade` and my local mysqldump is in ~fungi/wiki_2026-04-21_before_upgrade.sql` | 18:13 |
| @fungicide:matrix.org | sorry, missed a backtick... backup snapshot in rackspace is called `Wiki-MySQL-backup_2026-04-21_before_upgrade` and my local mysqldump is in `~fungi/wiki_2026-04-21_before_upgrade.sql` | 18:13 |
| @fungicide:matrix.org | starting the upgrade process | 18:13 |
| @fungicide:matrix.org | looks like it's nearly done | 18:25 |
| @fungicide:matrix.org | it did result in a new hostname, so i've updated the configs on the wiki server, checked that the new hostname works in mysqlclient (via the updated .my.cnf) and am rebooting the server now | 18:27 |
| @fungicide:matrix.org | the upgraded replica looked like it had the openstack_wiki db's tables, at least | 18:28 |
| @fungicide:matrix.org | https://wiki.openstack.org/wiki/Special:Version loads and reports `MySQL 5.7.34-log` | 18:30 |
| @clarkb:matrix.org | all good signs | 18:30 |
| @fungicide:matrix.org | i'm able to log in | 18:31 |
| @fungicide:matrix.org | looks like not completing the upgrade verification step resulted in the db coming up in read-only mode | 18:32 |
| @fungicide:matrix.org | i detacked the original from the replica and it's in the process of restarting | 18:32 |
| @fungicide:matrix.org | i'll reboot the wiki server again when that's done, for good measure | 18:32 |
| @clarkb:matrix.org | got it reading the email that effectively promotes the replica into the new standalone 5.7 server? | 18:33 |
| @fungicide:matrix.org | yes, it's apparently one-way replication, so the replica instance runs mysql in read-only | 18:34 |
| @clarkb:matrix.org | ya I guess that keeps the 5.6 around as a fallback too if necessary | 18:35 |
| @clarkb:matrix.org | until it gets force upgraded :) | 18:35 |
| @fungicide:matrix.org | okay, i was able to edit https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames to change the example review from 123456 to 654321 | 18:36 |
| @fungicide:matrix.org | so this seems fine | 18:36 |
| @clarkb:matrix.org | thats great | 18:36 |
| @clarkb:matrix.org | I'm approaching lunch time here, but I think we can probably proceed with cacti whenever you are feeling done with wiki. That way we can keep the momentum going and you don't even need to exit the dashboard :) | 18:37 |
| @clarkb:matrix.org | I see your edit and can view the diff of it too fwiw | 18:39 |
| @clarkb:matrix.org | and the timestamps lgtm | 18:39 |
| @fungicide:matrix.org | i do wonder, since the translate and translate-dev trove instances already running mysql 5.7 have an `_upgrade` appended to their names, like the new wiki instance does, if that means they might have been force-upgraded previously? | 18:40 |
| @fungicide:matrix.org | i see no snapshots for the old instances | 18:40 |
| @clarkb:matrix.org | maybe? | 18:41 |
| @clarkb:matrix.org | re cacti one thing I wanted to check on that I'm having a hard time understanding before we proceed is how are we limiting access. I don't see an obvious iptables rule and apache doesn't seem to be limtied to localhost | 18:41 |
| @clarkb:matrix.org | ok netstat does seem to indicate that apache is listening on localhost | 18:43 |
| @fungicide:matrix.org | `127.0.0.1:80` | 18:43 |
| @clarkb:matrix.org | so I want tofigure that out | 18:43 |
| @fungicide:matrix.org | yes, that's what i was just checking | 18:43 |
| @clarkb:matrix.org | its /etc/apache2/ports.conf | 18:44 |
| @clarkb:matrix.org | which is included by /etc/apache2/apache2.conf. So if we restart apache it should come up in a safe state. That is what i wanted to check and I'm happy for you to confirm | 18:45 |
| @fungicide:matrix.org | yes, that looks fine. i think for cacti we should be able to just perform the db upgrade, then switch over the config and restart apache to see what happens | 18:47 |
| @fungicide:matrix.org | i'm not even entirely sure we do writes to that database unless we're adding/removing configuration, do we? polled data is in whisper files right? | 18:48 |
| @clarkb:matrix.org | I'm not sure. I know that host info is in the db | 18:49 |
| @fungicide:matrix.org | the last entry in the `user_log` table was 3 years ago | 18:49 |
| @clarkb:matrix.org | but I'm not sure if anything liek snmp stuff ends up in there or not | 18:49 |
| @clarkb:matrix.org | for safety we can also just go ahead and stop apache if we want? | 18:49 |
| @clarkb:matrix.org | the db connection string appears to be in /etc/cacti/debian.php | 18:50 |
| @clarkb:matrix.org | and is unmanaged by puppet | 18:50 |
| @clarkb:matrix.org | fungi: we could 1. stop apache2 2. snapshot the db in trove and locally with mysqldump 3. upgrade the trove instance 4. update the cacti config to point at new db 5. start apache 6. browse 7. if all looks good promote the snapshot to primary | 18:51 |
| @fungicide:matrix.org | yeah, that's what i was going to edit, and `/root/.my.cnf` | 18:51 |
| @clarkb:matrix.org | oh yes ++ to editing that file afterwards as well | 18:51 |
| @fungicide:matrix.org | well, there wasn't any mysqlclient config until yesterday when i added one | 18:52 |
| @fungicide:matrix.org | since we didn't have mysqldump running in cron anyway | 18:52 |
| @fungicide:matrix.org | but yeah, the actual snmp data is stored as .rrd files in /var/lib/cacti/rra/ | 18:53 |
| @clarkb:matrix.org | so we may need to add a new host to see if writes to the db are working | 18:54 |
| @clarkb:matrix.org | now that I think of it I wonder if we missed that with some of mnasiadka's mirror swaps | 18:54 |
| @clarkb:matrix.org | so those may be a good test if general browsing works after the upgrade | 18:54 |
| @clarkb:matrix.org | and that isn't driven by puppet anymore it would need to be done by hand | 18:55 |
| @fungicide:matrix.org | okay, all ready for me to stop apache on cacti? | 18:55 |
| @clarkb:matrix.org | I think so | 18:56 |
| @fungicide:matrix.org | down it goes | 18:56 |
| @fungicide:matrix.org | making snapshot and dump now | 18:56 |
| @clarkb:matrix.org | looks like we add hosts to /var/lib/cacti/devices then a cronjob that runs at the top of every hour enrolls those hosts to cacti? | 18:57 |
| @clarkb:matrix.org | not sure if we need to disable the cronjob as part of this process but we are approaching the top of the hour | 18:57 |
| @fungicide:matrix.org | it's saved as `cacti-MySQL-backup_2026-04-21_before_upgrade` in rackspace and `~fungi/cacti_2026-04-20_before_upgrade.sql` on the server | 18:58 |
| @fungicide:matrix.org | shall i start the upgrade process? | 18:58 |
| @clarkb:matrix.org | fungi: do we want to awit for the cronjob to fire first? | 18:58 |
| @clarkb:matrix.org | it should start in a couple of minutes | 18:58 |
| @clarkb:matrix.org | but other than that I think we can proceed | 18:59 |
| @fungicide:matrix.org | yeah, can wait | 18:59 |
| @clarkb:matrix.org | seems to take a few seconds for each host according to ps output. SO probably a couple minutes before its done | 19:00 |
| @fungicide:matrix.org | watching `/var/log/cacti_update.log` | 19:01 |
| @jim:acmegating.com | no meeting today right? | 19:03 |
| @clarkb:matrix.org | corvus: correct | 19:03 |
| @jim:acmegating.com | if there were, i would have been on time. :) | 19:03 |
| @clarkb:matrix.org | we're doing mysql ops instead :) | 19:03 |
| @fungicide:matrix.org | reliving the golden age of the dba | 19:04 |
| @jim:acmegating.com | i'm semi-around for a bit; yell if you need me | 19:06 |
| @clarkb:matrix.org | ack and thanks | 19:06 |
| @fungicide:matrix.org | it's really been a non-event so far | 19:08 |
| @fungicide:matrix.org | though i'm probably jinxing us | 19:08 |
| @fungicide:matrix.org | looks like the register script finished | 19:11 |
| @fungicide:matrix.org | starting the db upgrade now | 19:11 |
| @fungicide:matrix.org | i updated the configs and confirmed the data is there on the upgraded db | 19:19 |
| @fungicide:matrix.org | starting apache again | 19:19 |
| @fungicide:matrix.org | note that the db is in read-only mode since i haven't detached the replica yet | 19:20 |
| @clarkb:matrix.org | I set up a port forward and can browse to graphs for at least one host | 19:20 |
| @fungicide:matrix.org | okay, i'll go ahead and detach the replica, which will cause the trove instance to restart, disconnecting the db socket briefly | 19:23 |
| @clarkb:matrix.org | ack | 19:23 |
| @fungicide:matrix.org | it's back up again, now in read/write mode | 19:26 |
| @clarkb:matrix.org | so I guess to fully exercise this we want to manually run that cron job command and then if that looks happy we rerun it again with a new host in the list? | 19:26 |
| @clarkb:matrix.org | corvus: ^ this is probably the bit where you might be able to help. That should exercise the db in a read write manner right? | 19:27 |
| @jim:acmegating.com | yep, that should make a bunch of new records | 19:27 |
| @jim:acmegating.com | i think we are missing hosts too | 19:27 |
| @jim:acmegating.com | so there should be some to add | 19:27 |
| @clarkb:matrix.org | corvus: ya we are because the cacti server has been in the emergency file. I'll find one that we can add | 19:27 |
| @clarkb:matrix.org | mirror.dfw3.raxflex.opendev.org is one we can add | 19:28 |
| @clarkb:matrix.org | fungi: ^ does the plan I proposed above seem reasonable? | 19:29 |
| @clarkb:matrix.org | if so maybe we start a root screen and run the cron job command without any new hosts first. Then add mirror.dfw3.raxflex.opendev.org to the /var/lib/cacti/devices list and rerun it? | 19:30 |
| @fungicide:matrix.org | yes, sounds great | 19:30 |
| @fungicide:matrix.org | i havea root screen session open on cacti02 | 19:31 |
| @clarkb:matrix.org | I have joined the root screen | 19:31 |
| @jim:acmegating.com | gitea-lb03 is not in there | 19:31 |
| @fungicide:matrix.org | i can add it now | 19:32 |
| @jim:acmegating.com | (just noting potential future hosts for this exercising) | 19:32 |
| @clarkb:matrix.org | if we want to run the comamnd as a noop we don't want to add it yet. Though I'm guessing it will fail the same way whether we add it or not | 19:32 |
| @clarkb:matrix.org | so maybe we just proceed with a new host in the list | 19:32 |
| @fungicide:matrix.org | it's added, though being all one giant line does make editing fun | 19:33 |
| @clarkb:matrix.org | ya its a fun little file. I think we can proceed now with the cron job command and see what happens | 19:34 |
| @fungicide:matrix.org | i've added a second window in the screen session tailing `/var/log/cacti_update.log` | 19:35 |
| @fungicide:matrix.org | the output looks the same as what i saw when watching it earlier pre-upgrade. i expect we won't get to the fun bits until it hits gitea-lb03 | 19:35 |
| @clarkb:matrix.org | it just said it added graphs for gitea-lb03 which si different to the noop I'm not doign anything because the thing arleady exist messages for the other hosts | 19:37 |
| @jim:acmegating.com | yep they exist now | 19:37 |
| @clarkb:matrix.org | so that is a good sign. I guess we have to wait ~5 minutes to see if it populates the rrd data, but I think from a db perspectiev that is a good indicator? | 19:37 |
| @jim:acmegating.com | yeah, it make take a few mins for the rrd data to show up -- maybe even 10 before we see real data | 19:37 |
| @fungicide:matrix.org | `Adding gitea-lb03.opendev.org (gitea-lb03.opendev.org) as "Linux Host" using SNMP v2 with community "public"` | 19:37 |
| @fungicide:matrix.org | `Success - new device-id: (733)` | 19:38 |
| @fungicide:matrix.org | followed by a lot of `Graph Added` lines | 19:38 |
| @fungicide:matrix.org | and eventually moves on to the next host, no errors reported at least | 19:38 |
| @clarkb:matrix.org | yup. So I guess the next step is to decide if/when we want to remove the old 5.6 databases for both wiki and cacti? They said they would start upgrades at 9am central tomorrow? So sometime today is probably a good idea? | 19:39 |
| @clarkb:matrix.org | as a general heads up I need to do the school run in about 1.5 hours. I missed the morning run which I typically do due to the PTG so should do the afternoon one | 19:40 |
| @fungicide:matrix.org | `select id from host where hostname="gitea-lb03.opendev.org";` | 19:40 |
| @fungicide:matrix.org | returns `733` | 19:41 |
| @fungicide:matrix.org | so that matches the log | 19:41 |
| @fungicide:matrix.org | yeah, i figure i can delete them later today. i'm leaving in about 90 minutes to meet some friends for dinner but could do it right before or else when i get home | 19:42 |
| @clarkb:matrix.org | sounds good | 19:44 |
| @jim:acmegating.com | (i think we have empty rrd files now for lb03, next pass should get data) | 19:44 |
| @fungicide:matrix.org | the script did finish, exiting 0 for what that's worth | 19:45 |
| @clarkb:matrix.org | excelletn I detached from screen | 19:45 |
| @fungicide:matrix.org | closing it down | 19:45 |
| @jim:acmegating.com | and now there's real data | 19:48 |
| @clarkb:matrix.org | great do we consider this done other than the old db cleanup steps? | 19:49 |
| @jim:acmegating.com | sgtm | 19:49 |
| @jim:acmegating.com | thanks Clark and fungi ! | 19:49 |
| @clarkb:matrix.org | I'm going to find lunch and then I have teh school run and then I think I have a mandatory training thing I need to get done before I'm traveling next week so that i odn't have to do it while traveling | 19:50 |
| @fungicide:matrix.org | yeah, seems done to me aside from me deleting those pre-upgrade instances in a bit | 19:50 |
| @fungicide:matrix.org | the trove instance hostname for wiki appeared in our private hostvars, so i updated it there just in case it gets reused | 20:24 |
| @fungicide:matrix.org | cacti's did not, nor do either of them appear in system-config so nothing to update there | 20:25 |
| @clarkb:matrix.org | Ya the cacti debian.php file doesn't appear in puppet at all as far as I can tell | 20:26 |
| @fungicide:matrix.org | okay, deleting the pre-upgrade mysql 5.6 databases for cacti and wiki now | 21:07 |
| @fungicide:matrix.org | all we've got left in that tenant now are mysql 5.7 based trove instances | 21:08 |
| @fungicide:matrix.org | okay, heading out now, will check back in an hour or two just in case any of this blows up | 21:11 |
| @clarkb:matrix.org | there was no confirmation from steve about gerrit account success in here as far as I can tell (but I've been busy maybe I overlooked it). I do see that email address has pushed a new python3.15 change to gerrit though so I think all is well there now | 21:42 |
| @fungicide:matrix.org | Clark: there was | 23:04 |
| @fungicide:matrix.org | at 00:56 utc today | 23:04 |
| @fungicide:matrix.org | Clark: https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2026-04-21.log.html#opendev.2026-04-21.log.html%23t2026-04-21T00:56:58 | 23:05 |
| @clarkb:matrix.org | Ah yup I completely missed that with everything else going on. Good to have that confirmation | 23:10 |
| @fungicide:matrix.org | and since nothing i did before dinner seems to have caught fire, i'm going to bow out and rest up for another day of ptg fun. night all! | 23:13 |
| @clarkb:matrix.org | Enjoy I've been winding down on my end too | 23:13 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!