| opendevreview | Ghanshyam proposed openstack/project-config master: Add separate ACL for openstackdocstheme https://review.opendev.org/c/openstack/project-config/+/940845 | 02:59 |
|---|---|---|
| frickler | jrosser: are you still using the held node? it's fine if you do, I just want to avoid it idling without being needed | 09:06 |
| jrosser | frickler: i think i can release that now - i understand enough to reproduce it locally now | 09:07 |
| frickler | jrosser: ok, removed it now, feel free to ping if you need a new one | 09:13 |
| *** dhill is now known as Guest8353 | 12:15 | |
| clarkb | if anyone has time to review the gitea upgrade change its a smallish bugfix update: https://review.opendev.org/c/opendev/system-config/+/940823 I can monitor it if approved | 15:33 |
| clarkb | there is also https://review.opendev.org/c/opendev/system-config/+/940350 to update the source of mariadb on the standalone mariadb we run for zuul. | 15:34 |
| clarkb | Thats the last mariadb source location update too I think | 15:35 |
| corvus | lgtm; do you want to do the zuul db upgrade now too? | 15:39 |
| corvus | s/upgrade/change/ :) | 15:40 |
| clarkb | I think so? the db location change hasn't been an issue yet if you accept the service restarts | 15:41 |
| corvus | +3 | 15:41 |
| clarkb | thanks | 15:43 |
| clarkb | the mailman db connection reset emails are a good indicator for whether or not we've pulled new mariadb images | 15:57 |
| clarkb | corvus: re ^ annoyingly quay shows the last modified timestamp as updating even when we don't change the image | 16:02 |
| clarkb | that seems like a quay db bug where they update a timestamp even if nothing else in the db is changing on push. But I wonder if we should try to detect that and noop in our jobs | 16:03 |
| clarkb | doing so might also potentially minimize the total number of docker hub requests improving our likelihood of success for images? | 16:03 |
| clarkb | the problem is I'm not sure what the best way to compare is as there are many different hashes and digests and they don't always align like I would expect them to | 16:04 |
| clarkb | I guess we would have to check per architecture digest as those should align? | 16:04 |
| corvus | i think there's a manifest that has all the arches in it... so even then there should be a single hash | 16:29 |
| clarkb | I think when we push the manfiest into quay we're pushing a revised manifest that only includes the arches we push | 16:31 |
| clarkb | so the manifest shas don't match, but we should be able to fetc hteh manifests themselves and see if there is overlap within them | 16:32 |
| clarkb | I think the gitea change will land after hourly jobs start so we're probably at least half an hour from that deploying even if it merges sooner | 16:50 |
| clarkb | we must've gotten a slow node I expected it to have merged by now | 17:18 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add separate ACL for openstackdocstheme https://review.opendev.org/c/openstack/project-config/+/940845 | 17:21 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Add separate ACL for openstackdocstheme https://review.opendev.org/c/openstack/project-config/+/940845 | 17:23 |
| opendevreview | Merged opendev/system-config master: Update to gitea 1.23.3 https://review.opendev.org/c/opendev/system-config/+/940823 | 17:25 |
| opendevreview | Merged opendev/system-config master: Switch standalone mariadb to opendevmirror hosted mariadb image https://review.opendev.org/c/opendev/system-config/+/940350 | 17:25 |
| clarkb | gitea deployment is starting. Also on reinspection I don't think we'll automatically update and restart mariadb on zuul-db01. I'll wait for the deployment to complete and confirm and if that is the case I can manually pull, down and up the container | 17:31 |
| clarkb | corvus: ^ fyi | 17:31 |
| corvus | clarkb: ack; if you don't feel like it or get around to it, i can do that later this evening. | 17:32 |
| clarkb | thanks I may wait a bit for a quieter zuul time at least (the check queue is a bit busy today for openstack) | 17:34 |
| clarkb | gitea09 has updated and looks good at https://gitea09.opendev.org:3081/opendev/system-config/ | 17:34 |
| clarkb | I'll start a git clone against that server too | 17:34 |
| clarkb | git clone seems to work | 17:35 |
| fungi | spot checking here, gitea09 is working for me | 17:37 |
| clarkb | upgrade is through 11 so far | 17:38 |
| clarkb | and now all 6 are done | 17:43 |
| clarkb | and the job reported success https://zuul.opendev.org/t/openstack/build/22997926c1b6454abd6a1d7b42bc1aa6 | 17:44 |
| clarkb | the zuul db deploy job is complete and docker image list and docker ps -a indicate that we didn't pull the new image and we did not restart the container | 17:46 |
| clarkb | this confirms we will need to manually update things there. As mentioned zuul is busyish now so I'll wait and look later. corvus I guess let me know if you're ready to do it and it hasn't been done yet that way we don't step on each other's toes | 17:47 |
| opendevreview | Merged openstack/project-config master: Add separate ACL for openstackdocstheme https://review.opendev.org/c/openstack/project-config/+/940845 | 18:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to grafana 11.5.1 https://review.opendev.org/c/opendev/system-config/+/940997 | 18:58 |
| clarkb | I fully expect that to fail but I figure its a quick way to start getting feedback from grafyaml against the new version | 18:59 |
| clarkb | but also this reminded me that I need to clean up grafana01. I haven't heard any objections to doing that so I'll proceed in the next 5 minute or so if no one speaks up with a concern | 18:59 |
| fungi | sgtm, thanks! | 19:00 |
| clarkb | #status log Deleted grafana01 (5ecfe52f-c2c3-44c8-b732-c4795f3b9ee6) as it has been replaced by grafana02 | 19:06 |
| opendevstatus | clarkb: finished logging | 19:06 |
| clarkb | neat grafana 11 succeeded in its deployment | 19:18 |
| fungi | whoa | 19:18 |
| clarkb | I guess the next step is to fetch the dashboard definitiosn and compare them to the 10.4.15 deployment and see how they differ | 19:18 |
| clarkb | I suspect that we're taking advantage of the automatic rewriting of angular stuff into not angular but having grafyaml write the correct data in the first place would be nice | 19:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to grafana 11.5.1 https://review.opendev.org/c/opendev/system-config/+/940997 | 19:35 |
| clarkb | If I haven't written bugs (I probably did) ^ should capture the json definitions for each dashboard | 19:36 |
| clarkb | then we can compare them to what we have in 10.4 | 19:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to grafana 11.5.1 https://review.opendev.org/c/opendev/system-config/+/940997 | 20:04 |
| clarkb | surprisingly the graph definitions don't seem to chagne at all? | 20:37 |
| clarkb | I've pulled the json definitions for afs and zuul-status form current server in prod and what the ci job against 11 gathered and some top level dashboard metadata changes like timestamps and versions and ids. But the graphs themselves are all still type: graph and the definitions don't differ | 20:37 |
| clarkb | the automatic conversion must be behind the scenes and not require updating of the graphs themselves | 20:38 |
| clarkb | its possible this would just work to upgrade to. Probably still worth holding a node and interacting with it more directly to try and figure this out more concretely though | 20:38 |
| clarkb | this genuinely surprised me beacuse I was expecting an annoying s///g type replacement but upstraem seems to be makign it easy for us | 20:39 |
| clarkb | "A dashboard must still be saved with the new plugin ID to persist the change." ya this statement has me thinking we need to hold a node and figure out how to force it to save in the new format | 20:59 |
| clarkb | but still maybe that isn't necessary. There aren't any warnings aynmore in the screenshots | 20:59 |
| clarkb | zuul never really got quiet like I expected it to. Anyway there are a couple of changes in the openstack gate queue that should finish up in the next 5-10 minutes then I'll start working on the database image update | 22:20 |
| clarkb | one change cleared out. Still waiting on the other. I'ev run docker-compose pull to stage the image then I'll do `docker-compose down && docker-compose up -d` | 22:29 |
| clarkb | once this other change clears out | 22:29 |
| clarkb | heh I forgot that merging things kicks off even mroe jobs... lets see what the estimates are on those | 22:44 |
| clarkb | I think these may be done pretty quickly then I'll proceed | 22:44 |
| clarkb | oh for some reason translation updates for ironic take 18 minutes? | 22:46 |
| clarkb | that seems odd | 22:46 |
| clarkb | ok all the post jobs but translation updates have finisehd. I'm proceeding | 22:52 |
| clarkb | its done. I am able to query for builds and open them | 22:54 |
| clarkb | the publish-openstack-python-branch-tarball job that finished at 22:49:58 February 7 was the last job to get recorded pre restart | 22:55 |
| clarkb | still waiting for something to finish after (like the translation job) and get recorded | 22:55 |
| clarkb | infra-prod-bootstrap-bridge that completed at 23:01:29 on February 7 is the first to get recorded as finishing after the db restart | 23:03 |
| clarkb | it shows up in the list and is then browseable (logs etc) | 23:03 |
| clarkb | oh sorry those are start times not end times | 23:04 |
| clarkb | corvus: just a thought I often thing end time is useful too? Not sure if we want to add that to the build listing | 23:04 |
| clarkb | in any case the restart doesn't appear to have been impactful from what I can see | 23:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!