Wednesday, 2023-10-11

opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
*** d34dh0r5- is now known as d34dh0r5313:05
*** amoralej_ is now known as amoralej13:23
opendevreviewClark Boylan proposed opendev/system-config master: Rebuild gitea 1.20.5 on latest golang 1.20.10
clarkbinfra-root: I've got a pile of image rebuilds to land starting with ^ then also (and children) and finally
opendevreviewClark Boylan proposed opendev/system-config master: Bump zookeeper from 3.7 to 3.8
clarkbWe will want to restart Gerrit on the new image just to be triple sure that it odesn't introduce any regressions. I'm happy to help babysit any and all of those changes today15:58
clarkb'Test cases: finished with 1 passing and 0 failing out of 1 test cases' huzzah17:06
clarkbI found another bug in the replication plugin that I had to fix to make my thing work. My thing is still probably broken though 17:07
fungipopping out to run an errand, should be back by 18z (likely well before)17:17
clarkbwhen you get back lets go ahead and approve the gitea update?17:18
fungiyou bet17:18
clarkband then we can do lodgeit and if we still have time today end with gerrit?17:18
fungisounds like a plan17:18
clarkbI think the gerrit update is the least important and most impactful hence its going last17:18
fungimakes sense. bbiab17:18
fungiclarkb: i'm back if you want me to start approving changes in the order you indicated17:43
clarkbfungi: will do17:44
clarkbjust going to double check the gitea screenshots now that check is done and will approve if they look good17:44
fungiah perfect. standing by to test once that deploys, otherwise continuing to work on prep for tomorrow's maintenance17:46
clarkbcorvus: gear's image update is ready if you are I think I captured what we agreed on in discussion but would appreciate if you can take a quick look to make sure it matches your expectations17:46
opendevreviewMerged opendev/gear master: Update gear to python3.11 and Bookworm container
opendevreviewJeremy Stanley proposed opendev/system-config master: Move OpenStack lists to Mailman 3
fungiinfra-root: that's ^ the change we need to merge tomorrow before maintenance begins18:09
clarkbfungi: did you cross check that no new lists were added to the old server since we added the commented out content? I don't remember any18:52
clarkbalso we're going to leave the old config in place for now on the old server and control dns/smtp/mm2 on the old side to keep it form functioning.18:53
clarkbnote the install languages might actually be a noop now that I see it18:53
clarkbI think mm3 just works better with different langauges because django18:53
fungiclarkb: yeah, nothing new in
fungiother than our migration changes18:54
clarkbthe gitea update should land shortly18:55
clarkbI was hoping it would get in before the hourly jobs but not sure it will18:55
clarkbthen once that is done I'll approve the three lodgeit changes and I think for that one I will manually pull and restart the service rather than wait for the daily job18:55
fungisounds great18:55
opendevreviewMerged opendev/system-config master: Rebuild gitea 1.20.5 on latest golang 1.20.10
opendevreviewJeremy Stanley proposed opendev/system-config master: Remove the old mailing list server
fungithat's ^ wip, for some time after the maintenance is over19:03
clarkball of the giteas have restarted19:11
clarkband they all respond to http requests so I think we are good19:11
clarkbI'll approve ldogeit changes now19:12
fungiyep, gitea servers lgtm19:13
fungiinfra-root: i've fleshed out the maintenance plan for tomorrow if anyone has time to give it a quick once-over: (starting at line #281)19:20
fungiother than waiting for the initial rsync of the ml content to the new server (which has been going for a couple of hours already), i think we're all set19:21
opendevreviewMerged opendev/lodgeit master: Put regex multiline specifier at beginning of regex string
opendevreviewMerged opendev/lodgeit master: Replace inspect.getargspec with inspect.signature
opendevreviewMerged opendev/lodgeit master: Update lodgeit container image to Python3.11 on Bookworm
fungithis failure is confusing me:
fungilooks like ansible's apt_repository module is choking on
fungilike maybe something's broken with the data serialized by _get_ppa_info19:41
fungi still seems to exist19:42
clarkbanyone else seeing high packet loss to review02? Seems to start for me in cogent19:59
clarkbmaking it really difficult to see which change hit that ppa thing19:59
fungi0% loss for me, over both ipv4 and ipv620:00
fungiclarkb: it's my "Remove the old mailing list server" patch20:00
clarkbfungi: fwiw it seems like it expects json and got something else? maybe figure out what request it is making and check if it is speaking json?20:01
fungiyeah, will probably need to track down the source for that ansible module and piece together whatever query it's making20:01
clarkbI cannot get that chaneg to render :/20:02
clarkbthanks, I don't see anything in there that would affect ppa stuff so ya probably need to figure out what ansible is doing20:04
fungipossible launchpad is just having a bad day20:05
clarkbcould be related to cogent being sad for me20:05
fungiall the other jobs for that change succeeded (except the non-voting ansible/devel job)20:05
clarkblooks like the promote job for 897711 did succeed according to zuul (couldn't check via gerrit) so I'm going to try the lodgeit restart now20:05
clarkb I was able to paste that paste20:07
fungiit's responding for me again20:08
fungiyeah, i pulled
clarkband rendered a lsit of functions20:08
clarkbcool I think this is working20:08
*** Guest2219 is now known as diablo_rojo20:09
fungii was able to create with the pastebinit tool20:09
fungiand with the web form20:10
fungiall good i think20:10
clarkbjust trying to triple check that it is running the image I want. `sudo docker ps -a` doesn't say latest next to the image name20:10
clarkbmaybe a column width thing20:11
fungidoes it take a -w flag?20:12
fungii don't see anything for wider output20:13
clarkbit takes a --no-trunc whcih gives me more info. I'm guessing latest is implied20:13
clarkbwhen it isn't tagged then you get the actual hash of the image20:13
clarkbdocker exec shows it is bookworm and python3.11 though so I think we are good20:13
fungioh, yep so that's what it meant20:13
fungiclarkb: i'm going to be heading out to dinner with friends in about half an hour, but will be home again at like 22:30 or maybe 23:00 utc if you wanted to wait to do the gerrit upgrade until i'm around20:14
clarkbfungi: ok should we go ahead and approve that change now then?20:14
fungiyeah, the restart is still manual20:14
fungiand then i'll ping you once i'm back and can help restart/test20:14
clarkbproblem: my network connectivity is really making this tough20:15
fungioh! right20:15
clarkbI'm not super confident I'll be able to even approve the change right now20:15
fungithat might be a good reason to postpone20:15
clarkbya I won't be able to help debug effectively if this keeps up and somethign goes wrong20:15
fungii could approve it, but if i'm the only one who can test things out, that may not be the wisest timing20:15
clarkbcogent ord is fine then whatever sbn is starts being sad20:16
clarkbsouth bend?20:17
fungiwould be nice to get it done, but it's not urgent. maybe tomorrow after mailing list maintenance is done and dusted we can revisit if your connection has recovered20:17
clarkbthere are a lot of hops in the midwest20:17
clarkbI have no idea how accurate that will be but it seems to line up with what I'm seeing20:18
clarkbfyi for anyone else having trouble getting to it may be packets getting lost in cogent20:20
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
fungiheading out, back in a couple hours20:33
clarkblooks like my route goes south now all the way to atlanta then up the eastern seaboard completely bypassing the midwest20:36
clarkbseems happier but I think too early to trust it for gerrit work20:36
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
opendevreviewGhanshyam proposed openstack/project-config master: End gate and update acl for retiring js-openstack-lib
opendevreviewJan Marchel proposed openstack/project-config master: Add NebulOuS repos: exn and gui
fungiokay, back22:32
fungiand yeah, not sure i'd trust cogent just yet if i were you22:32
fungibut i'm around if you want to do anything22:32
Clark[m]I took advantage for a bike ride. Give me a minute to see if connectivity is good and if so we can probably give it a go22:42
fungisure, i'm around22:44
clarkbI'm still going through the SE and connectivity looks good lets approve it I guess22:46
opendevreviewMerged opendev/system-config master: Update Gerrit to 3.7.5
clarkbfungi: plan is basically the same as last time: docker-compose pull, docker-compose down, mv waiting/ queue aside, docker-compose up -d23:33
clarkbthe image has promoted at this point and I'm raedy if you are23:34
clarkbopendevorg/gerrit   3.7       e1d8770d2496 <- current image23:34
clarkbhows this look #status log Another short Gerrit outage for updates on This update ensures we are using the current versions of all Gerrit plugins.23:37
clarkbI'm going to proceed with that shortly23:42
clarkb#status notice Another short Gerrit outage for updates on This update ensures we are using the current versions of all Gerrit plugins.23:45
opendevstatusclarkb: sending notice23:45
-opendevstatus- NOTICE: Another short Gerrit outage for updates on This update ensures we are using the current versions of all Gerrit plugins.23:45
fungiyep, looks like deployment succeeded23:45
clarkbI've pulled the new image but will wait for the bot to say it is done before stopping the service23:46
opendevstatusclarkb: finished sending notice23:47
clarkbok proceeding now23:48
clarkbI can get the web UI again23:49
clarkbthe reported jvm versions is still 17 so we didn't accidentally rollback via git merge errors (I didn't expect that but good to see all is well)23:49
fungiyep, loading for me and says 3.7.523:50
clarkbthe version of gerrit proper didn't change. But those two plugins I called out would've been updated by one commit each so ya a very minor update but a good one to get out of the way23:51
clarkbnext up: removing bullseye and bookworm python3.9 and python3.10 images23:52
clarkband then python3.12, except we don't have a good way to run unittests on that yet23:52
fungii'll keep half an eye out for concerns, but mostly retiring early so i can get things going for maintenance preliminary steps at 12:30 utc, a few hours ahead of the outage window23:55
clarkbI don't expect problems. I'll check in after dinner to make sure nothing went crazy23:55
fungiright on23:55
clarkband I should be around just after 1500 UTC tomorrow23:55

Generated by 2.17.3 by Marius Gedminas - find it at!