| @mnasiadka:matrix.org | Clark: not really, if the module sets no_log for these arguments and you don’t use shell/command - it’s fine | 04:20 |
|---|---|---|
| @mnasiadka:matrix.org | I have stopped old mirror servers (to wait another 24 hours before removing them) | 06:51 |
| - mirror02.dfw.rax.opendev.org (since we moved to mirror03 some time ago) | ||
| - mirror02.ord.rax.opendev.org | ||
| - mirror03.gra1.ovh.opendev.org | ||
| @mnasiadka:matrix.org | Uh oh, review03.opendev.org is in SHUTOFF - it seems it has been stopped with req-8d3a5247-592f-4cb9-b28d-684361bb26b4 whatever that means (cc: mnaser ) | 08:12 |
| @mnasiadka:matrix.org | infra-root: I started review03.opendev.org and started gerrit using `docker compose up -d` | 08:22 |
| @mnasiadka:matrix.org | Hopefully that's fine, although I've seen [DiskCache-Store-0] WARN com.google.gerrit.server.cache.h2.H2CacheImpl : Cannot put into cache jdbc:h2:file:///var/gerrit/cache/diff_summary-v2 and `org.h2.jdbc.JdbcBatchUpdateException: Timeout trying to lock table "DATA"` | 08:27 |
| @noonedeadpunk:matrix.org | folks, can you kindly dequeue the https://zuul.opendev.org/t/openstack/status?change=985276 ? As it looks like being stuck for some reason (maybe related somehow to gerrit issues?) | 11:37 |
| @noonedeadpunk:matrix.org | As I don't see any conflict tbh... | 11:37 |
| @noonedeadpunk:matrix.org | Or you mean it conflicts with https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/985319 | 11:38 |
| @noonedeadpunk:matrix.org | hm | 11:38 |
| @noonedeadpunk:matrix.org | ok, forget this, they're indeed conflict between each other | 11:39 |
| @clarkb:matrix.org | mnasiadka: I think starting the server then starting the service is the correct process there. Re the diff summary cache H2 v2 is apparently more sensitive to unexpected shutdowns than H2 v1 was and this is our first experience with what that looks like. I think we should check that the cache appears to be in use now (can check /home/gerrit2/review_site/cache timestamps maybe? Or maybe one of the cache info ssh commands will report info back). Mostly concerned we are not using that particular cache and any others that may not be lockable as a result of the shutdown | 13:59 |
| @clarkb:matrix.org | I think if we find they are not in use then shutting down and deleting the caches and letting them rebuild after starting Gerrit again is the process we would use | 13:59 |
| @clarkb:matrix.org | Then on the cloud side I want to say last time this happened it was due to memory oversubscription on the hypervisor and mnaser or someone else at vexxhost mitigated that | 14:00 |
| @clarkb:matrix.org | I can take a closer look in about an hour or so | 14:02 |
| @clarkb:matrix.org | Looking at timestamps for the caches the diff summary cache has updated as recently as a few minutes ago. Most of the caches have. There are a small number that haven't and I think that is beacuse we don't update them as often? Things like git tags, oauth tokens, groups, projects, and reverts | 15:01 |
| @clarkb:matrix.org | and looking at the error_log the db errors appear to have all clustered around the startup time. I notice that db pruning appears to have happened during that time period as well. I'm thinking that maybe the issue wasn't corrupted db files, but instead contention around db pruning during startup and operational tasks being unable to write during that time? | 15:01 |
| @clarkb:matrix.org | I ran `gerrit show-caches` over the ssh port and it did not error and the output looks mostly as I expect. The one thing that maybe looks wrong is that the git_tags cache says it has 0.00k on disk | 15:05 |
| @clarkb:matrix.org | maybe that changes if someone pushes a git tag? | 15:05 |
| @clarkb:matrix.org | But overall this seems like it is working. I would say keep eyes open for any unexpected behavior particularly around tags. And if we need to we can shutdown Gerrit intentionally, clear out any h2 db backing files that we suspect are a problem then start it up again. Note if you delete web_sessions-v2.mv.db everyone will need to log back in again afterwards | 15:06 |
| @fungicide:matrix.org | rackspace just sent us a notification that we have 5 trove databases running mysql 5.6 that will be upgraded automatically (disruptively) to 5.7 if we don't do so in the next two days | 15:10 |
| @clarkb:matrix.org | storyboard and zanata and wiki? I can't think of anything else that might still be there | 15:11 |
| @fungicide:matrix.org | some may be non-production | 15:11 |
| @fungicide:matrix.org | cacti is one, apparently | 15:14 |
| @clarkb:matrix.org | fungi: google says that the upgrade may impact queries using GROUP BY if they aren't constructed properly and tables using old TIMESTAMP or DATETIME columns will need to be repaired/rebuilt after the move | 15:14 |
| @clarkb:matrix.org | I see at least one group_by in storyboard's code base | 15:15 |
| @fungicide:matrix.org | fwiw, the wiki is using a local mysqld so i don't think either of the wiki trove instances they listed is in use | 15:16 |
| @clarkb:matrix.org | oh cacti is the other | 15:17 |
| @clarkb:matrix.org | so we probably need a list of things that will be affected and maybe we manually run mysqldumps if we aren't already doing so. And then hope for the best? I honestly don't feel like I'm in a position to try and correct any database issues with such an upgrade right now | 15:17 |
| @fungicide:matrix.org | storyboard and storyboard-dev are running local mysqld processes too, but not sure if they're using those | 15:18 |
| @clarkb:matrix.org | their connection strings are theoretically in the app config? that is probably the easiest way to check? | 15:19 |
| @fungicide:matrix.org | /etc/storyboard/storyboard.conf has `connection=mysql+pymysql://REDACTED:REDACTED@localhost:3306/storyboard?charset=utf8mb4` | 15:20 |
| @clarkb:matrix.org | ok so should be using the local db. We should identify what each of the reported DBs is from the rax dashboard, then cross check with services. But it seems like maybe cacti and zanata are things to worry about? | 15:21 |
| @fungicide:matrix.org | oh, while there is a local mysqld on the wiki server, /srv/mediawiki/Settings.php contains `$wgDBserver = "REDACTED.rackspaceclouddb.com";` | 15:22 |
| @fungicide:matrix.org | so it probably will be affected | 15:22 |
| @clarkb:matrix.org | basically if we get a human readable list of dbs from the rax side, then cross check against service configs we should get a complete list of things we should then backup. | 15:23 |
| @clarkb:matrix.org | and maybe we just do that and hope for the best because ya I'm not really feeling in a position to do more than that right now | 15:23 |
| @fungicide:matrix.org | funny, there's a local mysqld running on cacti too | 15:23 |
| @fungicide:matrix.org | i wonder if that's in use | 15:24 |
| @fungicide:matrix.org | oh yep, /etc/dbconfig-common/cacti.conf has `$database_hostname = "REDACTED.rackspaceclouddb.com";` | 15:25 |
| @clarkb:matrix.org | fungi: it is also possible some of those DBs won't be upgraded, but based on the email I suspect both cacti and wiki will be (but we should login and double check and confirm) | 15:25 |
| @fungicide:matrix.org | there's no mysqld nor mariadbd running on translate, so while i didn't check its config zanata probably is also affected | 15:26 |
| @clarkb:matrix.org | based on some cacti docs I'm hopeful that 5.7 won't be a big deal there | 15:31 |
| @clarkb:matrix.org | fungi: I think we do already have backups running for translate (good idea to double check they look like we expect in borg). I don't see backups for cacti and I need to use an old key for wiki I think that I don't have loaded so haven't checked that one | 15:33 |
| @clarkb:matrix.org | I need to finish breakfast things then jump into a PTG meeting in about 20 minutes. Maybe after (or during if we get bored) we start putting together an etherpad with the info we're collecting | 15:38 |
| @fungicide:matrix.org | yeah, i'm in ptg sessions until 18z and then planning to find something to eat, but can put something together after that | 15:40 |
| @clarkb:matrix.org | sounds good thanks | 15:40 |
| @clarkb:matrix.org | I suspect that the biggest risks are wiki and zanata simply because we can't easily upgrade them. We'd likely be left reconfiguring them to speak to a local db of the expected version if 5.7 doesn't work | 15:41 |
| @clarkb:matrix.org | cacti is a much simpler service with far less visibility, but also probably has the same fallback approach. So mostly we should make sure there isn't anything else that will be impacted, make sure we have backs for everything that is to make the fallback possible, and then ya take it from there I guess | 15:42 |
| @clarkb:matrix.org | and notes to track it all become important give the variety of things impacted and the differences in setup | 15:42 |
| @fungicide:matrix.org | translate01 has recent dumps in /var/backups/mysql_backups/ but they look empty | 15:44 |
| @clarkb:matrix.org | fungi: does it run the regular borg backup script? Maybe it switched to off host backups? | 15:44 |
| @fungicide:matrix.org | yes, it's running borg in crontab | 15:45 |
| @fungicide:matrix.org | and has mysql streams configured to dump the db at rackspace | 15:46 |
| @fungicide:matrix.org | same for wiki | 15:47 |
| @fungicide:matrix.org | cacti is not though, that i can tell | 15:48 |
| @clarkb:matrix.org | Ya cacti isn't. We can probably reduce our to-do list to confirm translate and wiki DB backups in borg look good. Manually run cacti DB backup. Check for any other services that might be affected | 15:51 |
| @clarkb:matrix.org | I'm finding some evidence that cacti won't work with 5.7 in Google searches. But it's hard to say definitively | 15:52 |
| @clarkb:matrix.org | https://forums.cacti.net/viewtopic.php?t=11262 specifically this | 15:53 |
| @clarkb:matrix.org | So we may need to edit the 5.7 server my.ini to set that SQL mode? | 15:54 |
| @fungicide:matrix.org | i was about to say we can just dump it to the local mysqld and switch to point there, but that server also has 5.7 installed | 15:55 |
| @clarkb:matrix.org | I wonder if it is worth asking if we can have more time? 2 days isn't much notice and comes at a very busy time for the two of us | 15:56 |
| @clarkb:matrix.org | The next ~3 weeks are busy busy | 15:56 |
| @fungicide:matrix.org | unrelated, i've got most of the important e-mail from the rackspace infra-root mailbox copied over to the gmail-hosted replacement, and am cleaning out folders which aren't being copied. the subfolder for the review@ address has 266k messages in it currently (i'm obviously not migrating those) | 16:13 |
| @clarkb:matrix.org | thanks I noticed the new account had more content in it this morning' | 16:13 |
| @clarkb:matrix.org | https://www.mediawiki.org/wiki/Compatibility#Database is the mediawiki compatbility matrix. Its not super clear how far + goes though | 16:36 |
| @clarkb:matrix.org | I don't really feel super confident in any of these tools talking to a newer database without changing the code or configs | 16:36 |
| @clarkb:matrix.org | https://hub.docker.com/layers/library/mysql/5.7-debian/images/sha256-8ba532f113f22d112fd3f8cbf9173c2a151c8699c968238cb0a6442818427fbf here is a mysql 5.7 on debian container image we can probably try to load the backups into to find any immediate problems? | 16:59 |
| @clarkb:matrix.org | Really I'm starting to feel the timing more than anything else is the issue here. We've gone from openstack release to ddos to gerrit upgrade to ptg and next week I have LF travel and then week after that i Think fungi is on vacation. I think it asking for a delay would be helpful from my perspective. But all three of these services have been on life support and if we have to accept brokeness for a bit maybe that is ok? | 17:02 |
| @jim:acmegating.com | yeah if cacti is briefly broken we'll survive | 17:03 |
| @clarkb:matrix.org | I just want a week where I can focus on what I thought was important on Friday and not the next fire :) | 17:04 |
| @fungicide:matrix.org | i'll note that moving wiki to a local db is probably not too hard, the mysqld already running on the server is 5.5 | 17:06 |
| @clarkb:matrix.org | ack | 17:06 |
| @clarkb:matrix.org | though I can't currenly ssh into that server (I think I need to load my old rsa key then I will be able to) | 17:06 |
| @clarkb:matrix.org | Just haven't had time to test that yet this morning | 17:06 |
| @fungicide:matrix.org | i can replace ssh public keys on there if anyone needs | 17:07 |
| @clarkb:matrix.org | I think the issue may be that my new key doesn't work on that old server? | 17:09 |
| @clarkb:matrix.org | I think I can solve this but I want to test it. I'll do that after PTG ends for the morning | 17:09 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921321: Add an opendev specific build of mediawiki https://review.opendev.org/c/opendev/system-config/+/921321 | 17:12 | |
| @mnasiadka:matrix.org | Clark: what's the next thing to pick up from the servers upgrade etherpad? I did some mini changes on https://review.opendev.org/c/opendev/system-config/+/921321 that should revive this stream a bit | 17:13 |
| @mnasiadka:matrix.org | (and I'm doing some work on the greptimedb long term storage for Prometheus, should be ready to upload tomorrow) | 17:14 |
| @clarkb:matrix.org | mnasiadka: zp02 getting a replacement should be straightforward as well as insecure-ci-registry as they are largely stateless applications. The registry relies on swift to store the data. And then other than cacti/prometheus I think updating the backup servers probably has tbe biggest impact but is likely to be a bit more involved. For the backup servers we would want to create new servers with new volumes and point backups to them. Then we can detach the existing volumes from the existing backup servers and attach them to the new backup servers so that we have that data for a time. Then we can remove the old backup servers | 17:16 |
| @clarkb:matrix.org | then in a year or similar time frame we can remove the old backup volumes from the new server and clean up the old volumes and rely only on the new volumes from that point fowrard | 17:16 |
| @mnasiadka:matrix.org | Ok, zp02 first - one thing at a time :) | 17:16 |
| @clarkb:matrix.org | works for me. All of this is a great help as it reduces the scope of the problem space | 17:17 |
| @mnasiadka:matrix.org | (and I need to remove the old mirror instances from OpenStack - but one hour after I stopped them gerrit went SHUTOFF) :) | 17:17 |
| @mnasiadka:matrix.org | (and remove the volumes of the old mirror nodes) | 17:17 |
| @fungicide:matrix.org | Clark: on wiki ssh, i have a `Host wiki.openstack.org` in my .ssh/config where i override the `IdentityFile` and also set `HostkeyAlgorithms +ssh-rsa` and `PubkeyAcceptedAlgorithms +ssh-rsa` | 17:18 |
| -@gerrit:opendev.org- Michal Nasiadka proposed on behalf of Tony Breeds: [opendev/system-config] 921322: Initial dump or mediawiki role and config https://review.opendev.org/c/opendev/system-config/+/921322 | 17:19 | |
| @clarkb:matrix.org | fungi: thanks. I forgot about the hostkey algorithm thing but that may also be useful here | 17:19 |
| @clarkb:matrix.org | fungi: I left some notes about static02-04 on https://etherpad.opendev.org/p/opendev-server-upgrade-planning since they were captured there | 17:22 |
| @clarkb:matrix.org | I don't think that is urgent, but maybe you can check that the notes I wrote make sense given the current state of things | 17:23 |
| @clarkb:matrix.org | Gerrit 3.13 added back a feature flag to disable teh AI button | 17:24 |
| @clarkb:matrix.org | so we can set that when we upgrade to 3.13 to avoid the confusion and hardcoded templates | 17:25 |
| @clarkb:matrix.org | I don't think it is in a release yet so our test image doesn't support it, But we could rebuild 3.13 from stable-3.13 or whatever if it comes to that | 17:25 |
| @mnasiadka:matrix.org | Clark: Any objections to remove the old mirror instances that I shut down in the morning? They have been removed from DNS and inventory last week, so I'd assume it's extra safe to do that. | 17:26 |
| @clarkb:matrix.org | no objections from me, but I haven't actively checked anything related to that this morning. I tried to double check things last week when I was approving the changes though | 17:27 |
| @mnasiadka:matrix.org | #status log Replaced mirror03.gra1.ovh.opendev.org (e4cf1ba1-4d0a-48d2-87b7-13899fa20128, volume 3768d0fc-ffb4-4c86-b094-1ae5ae595816) with mirror04.gra1.ovh.opendev.org | 17:29 |
| @status:opendev.org | @mnasiadka:matrix.org: finished logging | 17:29 |
| @clarkb:matrix.org | mnasiadka: I think one difference with zp02 and other non mirror nodes is that you should be updatingthe test node label in system-config for that nodes as you update them | 17:32 |
| @clarkb:matrix.org | whereas with the mirrosr we were already testing the variety of platforms we run on | 17:32 |
| @clarkb:matrix.org | thats a minor thing I just wanted to note it | 17:33 |
| @mnasiadka:matrix.org | #status log Replaced mirror02.ord.rax.opendev.org (4b16a662-761e-47a5-ada1-33256d4de297, volume 234c2b09-a2a9-4aba-b3d1-2bb7a0310f1f) with mirror03.ord.rax.opendev.org | 17:33 |
| @status:opendev.org | @mnasiadka:matrix.org: finished logging | 17:33 |
| @mnasiadka:matrix.org | Clark: ack | 17:33 |
| @mnasiadka:matrix.org | #status log Replaced mirror02.dfw.rax.opendev.org (7e608502-6e33-4a59-aa1a-b4e2257d0a11, volume 2172fe8e-3d0e-4f29-a2d1-ba12bd2cfb77) with mirror03.dfw.rax.opendev.org | 17:36 |
| @status:opendev.org | @mnasiadka:matrix.org: finished logging | 17:36 |
| @mnasiadka:matrix.org | Ok, mirrors done | 17:37 |
| @clarkb:matrix.org | awesome | 17:37 |
| @clarkb:matrix.org | fungi: I have managed to ssh into wiki now. I did just need the right key. I'm going to work on cleaning up old leaked images in rax classic as that was on my todo list from last week and it should hopefully be straightforward once I start working on it | 18:03 |
| @clarkb:matrix.org | but then after lunch I can probably help with filling out the db upgrade etherpad you volunteered to create | 18:03 |
| @fungicide:matrix.org | yeah, i'm about to pop out for food now that the ptg is on break | 18:07 |
| @clarkb:matrix.org | I've cleaned up rax-dfw and rax-ord of obviously leaked images. rax-iad is in progress and is a bit slower because there are more leaked images and some of them have duplicate names so I need to do a second pass hwere I use UUIDs instead of names | 18:35 |
| @clarkb:matrix.org | I also notice that there are maybe more zuu-launcher images than I expected so we may be leaking those as well | 18:36 |
| @clarkb:matrix.org | but I haven't looked into that angle yet. I'm focused on cleaning up the old nodepool images and the bionic images at the moment | 18:36 |
| @clarkb:matrix.org | #status log Cleaned up leaked Nodepool and Bionic images from rax classic cloud regions | 19:12 |
| @status:opendev.org | @clarkb:matrix.org: finished logging | 19:12 |
| @fungicide:matrix.org | thanks! | 19:31 |
| @fungicide:matrix.org | i'm catching back up and will then return to looking at our wiki database options, since that seems like the biggest potential impact | 19:32 |
| @fungicide:matrix.org | strangely, the translate database indicates it's 5.7 already | 19:54 |
| @fungicide:matrix.org | the wiki db does say it's 5.6 though, as does cacti | 19:55 |
| @fungicide:matrix.org | i resized the volume for the translate db from 6gb to 7gb since it was 90% utilized | 19:56 |
| @fungicide:matrix.org | the wiki server does have current local dumps of the trove db, fwiw | 19:57 |
| @clarkb:matrix.org | fungi: maybe translate trove DB is not one in the list they shared? We probably do need to login and check the specific list against our services. But good to know one less service to worry about likely | 19:59 |
| @fungicide:matrix.org | yeah, i'm logged in and see only 5 trove instances in total across all regions | 20:00 |
| @clarkb:matrix.org | They had DB names in the email are all five matching the 5 from the email? If so weird that the trove DB shows up. Probably worth a follow up to indicate that one appears to already be 5.7? And maybe even ask for a bit more time? | 20:01 |
| @fungicide:matrix.org | i do see random reports across the internet of people running mediawiki 1.28 with mysql 5.7 | 20:03 |
| @clarkb:matrix.org | It could be that cacti is the only real concern and presumably with the SQL mode workaround I found that will work too | 20:04 |
| @traylenator:matrix.org | > <@traylenator:matrix.org> Thanks for the reply. That sounds plausible. I tried a few weeks back and could not get in for a different error. One of the things I tried then was deleting and re-creating in U.1. | 20:07 |
| > Deleting/disabling/updating the old id in gerrit sounds correct . Whenever you or someone gets a chance. | ||
| > | ||
| Clark: If you are able to sort my account that would be great, I have half a dozen simple py3.15a8 fixes from the openstack on rawhide I want to submit. | ||
| @fungicide:matrix.org | Clark: oh! the e-mail from rackspace was html-only, and my mail client is configured to use w3m to conver it to plaintext. i guess the list of affected databases was supposed to be a table of uuids and names, so actually it's only 3 it just looked like 5 the way it came out | 20:11 |
| @fungicide:matrix.org | cacti, wiki-dev (which is unused), and wiki | 20:11 |
| @fungicide:matrix.org | so that matches what i'm finding in their dashboard | 20:12 |
| @clarkb:matrix.org | Oh it looked like 5 to me too in thunderbird | 20:13 |
| @clarkb:matrix.org | > <@traylenator:matrix.org> Clark: If you are able to sort my account that would be great, I have half a dozen simple py3.15a8 fixes from the openstack on rawhide I want to submit. | 20:13 |
| I can probably take a look in a bit. Are you ok with the old account being disabled so that a new account can be created? | ||
| @fungicide:matrix.org | Clark: okay, even in the ticket system it's spread over 5 lines, but looks like the three uuids listed there correspond to the three names listed (i just manually compared them all) | 20:16 |
| @fungicide:matrix.org | i made an on-demand backup of the wiki-dev-mysql trove instance called "Wiki-Dev-MySQL-backup_2026-04-20_before_deletion" and am deleting it, mainly just a safety measure though i'm quite certain the server i was using that with has been gone for years already | 20:25 |
| @fungicide:matrix.org | so that gets us down to two | 20:25 |
| @fungicide:matrix.org | since i see indication that people elsewhere are successfully running the same mediawiki version as us on mysql 5.7 i'm inclined to just upgrade and then deal with the fallout if something goes sideways, since we do have backups | 20:26 |
| @clarkb:matrix.org | fungi: thats not the db that the current wiki server talks to? | 20:27 |
| @fungicide:matrix.org | correct | 20:27 |
| @clarkb:matrix.org | I guess the instance is wiki dev but it talk to the prod db? | 20:27 |
| @clarkb:matrix.org | and re that plan I guess that works. It probably works for cacti too given the thread I found? | 20:27 |
| @fungicide:matrix.org | yeah, i had a series of wiki-dev servers i was using when trying to get the configuration management implemented years ago | 20:27 |
| @fungicide:matrix.org | i deleted the last wiki-dev server but not the test copy of the database i'd been using with it | 20:28 |
| @clarkb:matrix.org | got it | 20:28 |
| @fungicide:matrix.org | the production database instance is "Wiki-MySQL" and that's still intact | 20:28 |
| @clarkb:matrix.org | given that we think this may be a bit less of a crisis than perviously anticipated I guess I'll context switch to cleaning up Steve Traylen's old gerrit account. I've found the ID for the old account in Gerrit's error_log and found my notes from the last time I had to do this. | 20:31 |
| @clarkb:matrix.org | It will probably take a bit to page it all back in but I think I have what I need to do the necessary cleanups | 20:31 |
| @clarkb:matrix.org | I actually think this may be a simpler cleanup as it looks like maybe the preferred email address for the old account is not the conflicting email? In that case I can probably leave the old account as is and simply remove the conflicting email address. But I'm inferring this through the web ui for now. need to double check in the notedb and via api requests | 20:32 |
| @clarkb:matrix.org | Steve Traylen: I think the necessary cleanup is done now | 20:51 |
| @clarkb:matrix.org | note I didn't disable the old account since in this case the conflict only appeared to be as an extra email address. I only removed that one conflict from the old account | 20:51 |
| @clarkb:matrix.org | infra-root: I'm working on putting my notes together and will stash them on review03 like I have been doing | 20:51 |
| @clarkb:matrix.org | the log file is now at review03:/home/clarkb/gerrit_user_cleanups/logs/ in a file with todays date as a prefix | 20:56 |
| @clarkb:matrix.org | I have deescalated my privs now too. I think that concludes this cleanup assuming Steve Traylen is able to log in now | 20:57 |
| @clarkb:matrix.org | fungi: oh we don't have cacti backups though. Should we try to do one of those locally or via the trove backup system like you did with wiki? | 20:57 |
| @clarkb:matrix.org | fungi: I figure I'll defer to you on that since you've just been doing it for wiki, but I think we should have a cacti backup just in case | 20:58 |
| @fungicide:matrix.org | oh, sure i can do that, can probably also make a local dump to the server for convenience | 20:59 |
| @clarkb:matrix.org | that would be great, thank you | 21:00 |
| @fungicide:matrix.org | it's being created as `cacti-MySQL-backup_2026-04-20_before_upgrade` | 21:01 |
| @fungicide:matrix.org | and done | 21:01 |
| @clarkb:matrix.org | is that within rax or locally? | 21:02 |
| @fungicide:matrix.org | within rax. working on the mysqldump now | 21:02 |
| @clarkb:matrix.org | got it | 21:02 |
| @fungicide:matrix.org | i dumped it to ~fungi/2026-04-20.sql.gz | 21:10 |
| @fungicide:matrix.org | should i go ahead and upgrade in trove? | 21:10 |
| @clarkb:matrix.org | you mean now rather than waiting for wednesday? | 21:10 |
| @fungicide:matrix.org | yeah, as in do we want to be in control of when the upgrade happens, so we can check that things are still working? | 21:11 |
| @clarkb:matrix.org | I don't strongly object, but I'm not in a great spot to debug it if it goes sideways right now. I promised I would get some other stuff done today and I'm running out of time for that so will context switch shortly | 21:11 |
| @clarkb:matrix.org | maybe we want to do that tomorrow isntead? | 21:11 |
| @fungicide:matrix.org | yeah that's fine. i can always do new backups tomorrow too | 21:11 |
| @fungicide:matrix.org | now that we've worked out the details | 21:11 |
| @clarkb:matrix.org | ya I think I'll be able to juggle paying attention to that tomorrow morning after ptg stuff (which is earlier for me tomorrow than it was today) | 21:12 |
| @fungicide:matrix.org | the database dumps for wiki and cacti don't take long at all | 21:12 |
| @fungicide:matrix.org | same for backup snapshots in rackspace | 21:12 |
| @fungicide:matrix.org | so i could do new backups of each one right before upgrading, in case we need to fall back on them | 21:13 |
| @clarkb:matrix.org | great. Does your ptg schedule go late tomorrow? I think mine ends at 1500 UTC | 21:13 |
| @clarkb:matrix.org | the email had a suggested upgrade process too. We can decide if we want to follow that | 21:13 |
| @clarkb:matrix.org | oh inclusivity wg is 16-1700 | 21:14 |
| @clarkb:matrix.org | so maybe a bit later than I thought but not as late as today | 21:14 |
| @fungicide:matrix.org | yeah, i was about to say i have that, but am free the hour before and after | 21:14 |
| @clarkb:matrix.org | the hour prior we have our meeting :) so ya lets aim for ~1700 UTC then | 21:15 |
| @fungicide:matrix.org | ah yep, for some reason my mind blocks out memory of all other scheduled meetings during the ptg | 21:29 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!