| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/897881 | 02:46 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/897881 | 12:59 |
| *** d34dh0r5- is now known as d34dh0r53 | 13:05 | |
| *** amoralej_ is now known as amoralej | 13:23 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rebuild gitea 1.20.5 on latest golang 1.20.10 https://review.opendev.org/c/opendev/system-config/+/897984 | 15:54 |
| clarkb | infra-root: I've got a pile of image rebuilds to land starting with ^ then also https://review.opendev.org/c/opendev/system-config/+/895522 https://review.opendev.org/c/opendev/lodgeit/+/897711 (and children) and finally https://review.opendev.org/c/opendev/gear/+/895968 | 15:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Bump zookeeper from 3.7 to 3.8 https://review.opendev.org/c/opendev/system-config/+/897985 | 15:57 |
| clarkb | We will want to restart Gerrit on the new image just to be triple sure that it odesn't introduce any regressions. I'm happy to help babysit any and all of those changes today | 15:58 |
| clarkb | 'Test cases: finished with 1 passing and 0 failing out of 1 test cases' huzzah | 17:06 |
| clarkb | I found another bug in the replication plugin that I had to fix to make my thing work. My thing is still probably broken though | 17:07 |
| fungi | popping out to run an errand, should be back by 18z (likely well before) | 17:17 |
| clarkb | when you get back lets go ahead and approve the gitea update? | 17:18 |
| fungi | you bet | 17:18 |
| clarkb | and then we can do lodgeit and if we still have time today end with gerrit? | 17:18 |
| fungi | sounds like a plan | 17:18 |
| clarkb | I think the gerrit update is the least important and most impactful hence its going last | 17:18 |
| fungi | makes sense. bbiab | 17:18 |
| fungi | clarkb: i'm back if you want me to start approving changes in the order you indicated | 17:43 |
| clarkb | fungi: will do | 17:44 |
| clarkb | just going to double check the gitea screenshots now that check is done and will approve if they look good | 17:44 |
| clarkb | done | 17:45 |
| fungi | ah perfect. standing by to test once that deploys, otherwise continuing to work on prep for tomorrow's maintenance | 17:46 |
| clarkb | corvus: gear's image update is ready if you are https://review.opendev.org/c/opendev/gear/+/895968 I think I captured what we agreed on in discussion but would appreciate if you can take a quick look to make sure it matches your expectations | 17:46 |
| corvus | lgtm | 17:48 |
| clarkb | tyty | 17:52 |
| opendevreview | Merged opendev/gear master: Update gear to python3.11 and Bookworm container https://review.opendev.org/c/opendev/gear/+/895968 | 17:56 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Move OpenStack lists to Mailman 3 https://review.opendev.org/c/opendev/system-config/+/897998 | 18:09 |
| fungi | infra-root: that's ^ the change we need to merge tomorrow before maintenance begins | 18:09 |
| clarkb | fungi: did you cross check that no new lists were added to the old server since we added the commented out content? I don't remember any | 18:52 |
| clarkb | also we're going to leave the old config in place for now on the old server and control dns/smtp/mm2 on the old side to keep it form functioning. | 18:53 |
| clarkb | note the install languages might actually be a noop now that I see it | 18:53 |
| clarkb | I think mm3 just works better with different langauges because django | 18:53 |
| fungi | clarkb: yeah, nothing new in https://opendev.org/opendev/system-config/commits/branch/master/inventory/service/host_vars/lists.openstack.org.yaml | 18:54 |
| fungi | other than our migration changes | 18:54 |
| clarkb | the gitea update should land shortly | 18:55 |
| clarkb | I was hoping it would get in before the hourly jobs but not sure it will | 18:55 |
| clarkb | then once that is done I'll approve the three lodgeit changes and I think for that one I will manually pull and restart the service rather than wait for the daily job | 18:55 |
| fungi | sounds great | 18:55 |
| opendevreview | Merged opendev/system-config master: Rebuild gitea 1.20.5 on latest golang 1.20.10 https://review.opendev.org/c/opendev/system-config/+/897984 | 18:57 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove the old mailing list server https://review.opendev.org/c/opendev/system-config/+/898004 | 19:03 |
| fungi | that's ^ wip, for some time after the maintenance is over | 19:03 |
| clarkb | all of the giteas have restarted | 19:11 |
| clarkb | and they all respond to http requests so I think we are good | 19:11 |
| clarkb | I'll approve ldogeit changes now | 19:12 |
| fungi | yep, gitea servers lgtm | 19:13 |
| fungi | thanks! | 19:13 |
| fungi | infra-root: i've fleshed out the maintenance plan for tomorrow if anyone has time to give it a quick once-over: https://etherpad.opendev.org/p/mm3migration (starting at line #281) | 19:20 |
| fungi | other than waiting for the initial rsync of the ml content to the new server (which has been going for a couple of hours already), i think we're all set | 19:21 |
| opendevreview | Merged opendev/lodgeit master: Put regex multiline specifier at beginning of regex string https://review.opendev.org/c/opendev/lodgeit/+/897713 | 19:24 |
| opendevreview | Merged opendev/lodgeit master: Replace inspect.getargspec with inspect.signature https://review.opendev.org/c/opendev/lodgeit/+/897714 | 19:24 |
| opendevreview | Merged opendev/lodgeit master: Update lodgeit container image to Python3.11 on Bookworm https://review.opendev.org/c/opendev/lodgeit/+/897711 | 19:24 |
| fungi | this failure is confusing me: https://zuul.opendev.org/t/openstack/build/3b2fb33376df454d927148c8dc017cc1/console#1/0/3/ubuntu-xenial | 19:34 |
| fungi | looks like ansible's apt_repository module is choking on https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-pip/tasks/xenial.yaml#L4-L7 | 19:40 |
| fungi | like maybe something's broken with the data serialized by _get_ppa_info | 19:41 |
| fungi | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/python-pip/+packages still seems to exist | 19:42 |
| clarkb | anyone else seeing high packet loss to review02? Seems to start for me in cogent | 19:59 |
| clarkb | making it really difficult to see which change hit that ppa thing | 19:59 |
| fungi | 0% loss for me, over both ipv4 and ipv6 | 20:00 |
| fungi | clarkb: it's my "Remove the old mailing list server" patch | 20:00 |
| clarkb | fungi: fwiw it seems like it expects json and got something else? maybe figure out what request it is making and check if it is speaking json? | 20:01 |
| fungi | yeah, will probably need to track down the source for that ansible module and piece together whatever query it's making | 20:01 |
| clarkb | I cannot get that chaneg to render :/ | 20:02 |
| fungi | clarkb: https://opendev.org/opendev/system-config/commit/bbbb67a | 20:03 |
| clarkb | thanks, I don't see anything in there that would affect ppa stuff so ya probably need to figure out what ansible is doing | 20:04 |
| fungi | possible launchpad is just having a bad day | 20:05 |
| clarkb | could be related to cogent being sad for me | 20:05 |
| fungi | all the other jobs for that change succeeded (except the non-voting ansible/devel job) | 20:05 |
| clarkb | looks like the promote job for 897711 did succeed according to zuul (couldn't check via gerrit) so I'm going to try the lodgeit restart now | 20:05 |
| clarkb | https://paste.opendev.org/show/b5rqobZIwifmU6jAX1BM/ I was able to paste that paste | 20:07 |
| fungi | it's responding for me again | 20:08 |
| fungi | yeah, i pulled https://paste.opendev.org/show/bc0aOdtjAGq4lK3ySlnr/ | 20:08 |
| clarkb | and https://paste.opendev.org/help/api/ rendered a lsit of functions | 20:08 |
| clarkb | cool I think this is working | 20:08 |
| *** Guest2219 is now known as diablo_rojo | 20:09 | |
| fungi | i was able to create https://paste.opendev.org/show/bRwI8SnGrLB7WtG1SPdR/ with the pastebinit tool | 20:09 |
| fungi | and https://paste.opendev.org/show/bYT4LLlQrHHZss3VgKYj/ with the web form | 20:10 |
| fungi | all good i think | 20:10 |
| clarkb | just trying to triple check that it is running the image I want. `sudo docker ps -a` doesn't say latest next to the image name | 20:10 |
| clarkb | maybe a column width thing | 20:11 |
| fungi | does it take a -w flag? | 20:12 |
| fungi | i don't see anything for wider output | 20:13 |
| clarkb | it takes a --no-trunc whcih gives me more info. I'm guessing latest is implied | 20:13 |
| clarkb | when it isn't tagged then you get the actual hash of the image | 20:13 |
| clarkb | docker exec shows it is bookworm and python3.11 though so I think we are good | 20:13 |
| fungi | oh, yep so that's what it meant | 20:13 |
| fungi | clarkb: i'm going to be heading out to dinner with friends in about half an hour, but will be home again at like 22:30 or maybe 23:00 utc if you wanted to wait to do the gerrit upgrade until i'm around | 20:14 |
| clarkb | fungi: ok should we go ahead and approve that change now then? | 20:14 |
| fungi | yeah, the restart is still manual | 20:14 |
| fungi | and then i'll ping you once i'm back and can help restart/test | 20:14 |
| clarkb | problem: my network connectivity is really making this tough | 20:15 |
| fungi | oh! right | 20:15 |
| clarkb | I'm not super confident I'll be able to even approve the change right now | 20:15 |
| fungi | that might be a good reason to postpone | 20:15 |
| clarkb | ya I won't be able to help debug effectively if this keeps up and somethign goes wrong | 20:15 |
| fungi | i could approve it, but if i'm the only one who can test things out, that may not be the wisest timing | 20:15 |
| clarkb | cogent ord is fine then whatever sbn is starts being sad | 20:16 |
| clarkb | south bend? | 20:17 |
| fungi | would be nice to get it done, but it's not urgent. maybe tomorrow after mailing list maintenance is done and dusted we can revisit if your connection has recovered | 20:17 |
| clarkb | there are a lot of hops in the midwest | 20:17 |
| clarkb | wfm | 20:17 |
| clarkb | https://downdetector.com/status/cogent/map/ | 20:18 |
| clarkb | I have no idea how accurate that will be but it seems to line up with what I'm seeing | 20:18 |
| fungi | ick | 20:19 |
| clarkb | fyi for anyone else having trouble getting to review.opendev.org it may be packets getting lost in cogent | 20:20 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 20:29 |
| fungi | heading out, back in a couple hours | 20:33 |
| clarkb | looks like my route goes south now all the way to atlanta then up the eastern seaboard completely bypassing the midwest | 20:36 |
| clarkb | seems happier but I think too early to trust it for gerrit work | 20:36 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 20:37 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 21:00 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 21:11 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 21:29 |
| opendevreview | Ghanshyam proposed openstack/project-config master: End gate and update acl for retiring js-openstack-lib https://review.opendev.org/c/openstack/project-config/+/898015 | 21:58 |
| opendevreview | Jan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui https://review.opendev.org/c/openstack/project-config/+/898023 | 22:12 |
| fungi | okay, back | 22:32 |
| fungi | and yeah, not sure i'd trust cogent just yet if i were you | 22:32 |
| fungi | but i'm around if you want to do anything | 22:32 |
| Clark[m] | I took advantage for a bike ride. Give me a minute to see if connectivity is good and if so we can probably give it a go | 22:42 |
| fungi | sure, i'm around | 22:44 |
| clarkb | I'm still going through the SE and connectivity looks good lets approve it I guess | 22:46 |
| clarkb | done | 22:47 |
| fungi | cool | 22:52 |
| opendevreview | Merged opendev/system-config master: Update Gerrit to 3.7.5 https://review.opendev.org/c/opendev/system-config/+/895522 | 23:31 |
| clarkb | fungi: plan is basically the same as last time: docker-compose pull, docker-compose down, mv waiting/ queue aside, docker-compose up -d | 23:33 |
| clarkb | the image has promoted at this point and I'm raedy if you are | 23:34 |
| clarkb | opendevorg/gerrit 3.7 e1d8770d2496 <- current image | 23:34 |
| clarkb | hows this look #status log Another short Gerrit outage for updates on review.opendev.org. This update ensures we are using the current versions of all Gerrit plugins. | 23:37 |
| clarkb | I'm going to proceed with that shortly | 23:42 |
| clarkb | s/log/notice/ | 23:43 |
| clarkb | #status notice Another short Gerrit outage for updates on review.opendev.org. This update ensures we are using the current versions of all Gerrit plugins. | 23:45 |
| opendevstatus | clarkb: sending notice | 23:45 |
| -opendevstatus- NOTICE: Another short Gerrit outage for updates on review.opendev.org. This update ensures we are using the current versions of all Gerrit plugins. | 23:45 | |
| fungi | yep, looks like deployment succeeded | 23:45 |
| clarkb | I've pulled the new image but will wait for the bot to say it is done before stopping the service | 23:46 |
| opendevstatus | clarkb: finished sending notice | 23:47 |
| clarkb | ok proceeding now | 23:48 |
| clarkb | I can get the web UI again | 23:49 |
| clarkb | the reported jvm versions is still 17 so we didn't accidentally rollback via git merge errors (I didn't expect that but good to see all is well) | 23:49 |
| fungi | yep, loading for me and says 3.7.5 | 23:50 |
| clarkb | the version of gerrit proper didn't change. But those two plugins I called out would've been updated by one commit each so ya a very minor update but a good one to get out of the way | 23:51 |
| fungi | right | 23:51 |
| clarkb | next up: removing bullseye and bookworm python3.9 and python3.10 images | 23:52 |
| clarkb | and then python3.12, except we don't have a good way to run unittests on that yet | 23:52 |
| fungi | i'll keep half an eye out for concerns, but mostly retiring early so i can get things going for maintenance preliminary steps at 12:30 utc, a few hours ahead of the outage window | 23:55 |
| clarkb | I don't expect problems. I'll check in after dinner to make sure nothing went crazy | 23:55 |
| fungi | right on | 23:55 |
| clarkb | and I should be around just after 1500 UTC tomorrow | 23:55 |
| fungi | aswesome | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!