Thursday, 2023-08-17

opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
opendevreviewMerged opendev/system-config master: Setup #openstack-charms IRC channel.
fungilooks like is failing to renew the cert for insecure-ci-registry01 which expires today02:39
fungihuh, though that's no longer in dns, so why is it still trying to renew it?03:07
fungiinsecure-ci-registry02 is in dns03:07
Clark[m]fungi: because it is still in the inventory. It's been on my list to delete it. I have a change up to pull it from inventory first that hasn't gotten any reviews yet03:22
Clark[m]We can put it in the emergency file instead but I was hoping for the proper removal03:22
fungioh, i should have looked for that before i started writing one!03:25
fungiyours looks identical to what i was about to push03:26
opendevreviewMerged opendev/system-config master: Remove old insecure-ci-registry01 from our inventory
*** dasm is now known as Guest914809:01
opendevreviewStephen Finucane proposed openstack/project-config master: Add github sphinx-doc/sphinx to the project list
opendevreviewMerged openstack/project-config master: Add github sphinx-doc/sphinx to the project list
TheJuliaWould you awesome folks please put a job hold in for ironic-tempest-partition-uefi-ipmi-pxe-ovn  I'm trying to figure out why I'm getting no logging out of zuul for OVN so I can troubleshoot the configuration and having no luck14:16
fungiTheJulia: for what repository?14:18
TheJuliachange 88508714:19
TheJuliamuch appreciated, thanks14:19
fungiClark[m]: are you done with the autohold for etherpad 1.9.1 testing now that we've upgraded?14:20
fungifrickler: are you still using the autohold for debugging dynamic routing?14:21
fricklerfungi: yes14:21
fungithanks, i'll clean that one up14:22
fricklerehm, still using means please don't clean it14:22
fungioh, sorry :/14:22
Clark[m]fungi: yes the etherpad 1.9.1 hold can be deleted. They just released 1.9.2 today too :)14:22
fungifrickler: apologies, i'll add you a new autohold, i missed that i asked you if you were using it rather than done with it. i'll add another hold14:23
fricklerfungi: nevermind, I was mostly done, I guess I can also go on without it14:24
fungiokay, really sorry about that14:25
fungiClark[m]: etherpad hold is gone now14:25
fungiClark[m]: once you're more awake, did you want to look closer at the possibility of a gitea upgrade before the weekend?14:37
fungialso i think i got the mailman 3 restart handler updated with your recommendation14:38
Clark[m]Re the gitea upgrade I think the main thing is general review of the existing change and some thought on whether or not we care about configuring a bunch of oauth2 settings we ultimately end up disabling.14:50
fungisounds good14:51
Clark[m]I'll take a look at the mm3 change shortly.15:03
*** Guest9148 is now known as dasm15:11
clarkbfungi: yup that looks correct. I half wonder if the condition in the handler is worth a comment but I've +2'd it.15:30
fungithere's no rush, but presumably we'd also need to merge another config change to actually trigger the restart, right?15:30
fungioh, also i can certainly add a comment there15:31
fungioh, or i can fudge a config change by manually rolling back the SITE_ID update, right?15:31
clarkbthinking about gitea, I don't think I'm in a rush there either and would prefer reviewers look over it carefully. This is the biggest gitea upgrade in terms of moving pieces on our side (but not features, weird how that works out) so adding confidence to it would be nice15:31
clarkbfungi: any update to the files that notify the handler should do it15:32
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
clarkbtherefore for gitea I think its best if we focus on review for now and then decide when we're comfortable merging/deploying that change after reviews15:34
fungisure, that seems like a safe way to proceed15:35
fungiso if you want to approve 891555 with the added comment, i'll manually switch SITE_ID back to 1 on the server and then when the change rolls out we should see the deploy job correct the value in back to 0 and then restart the containers after doing so15:37
fungiat which point the server name on the archive pages should start to have the correct string15:38
fungii changed the value in /var/lib/mailman/web/ at 15:39 utc (what's now showing as the last modified timestamp)15:40
clarkbfungi: that change also removes the inventory file for insecure ci registry. was that intentional?15:48
clarkbI think git may handle that properly as long as it is identical to the change that landed yesterday15:48
fungioh, no that was not. that was the mostly written change i was about to push last night and i forgot to hard reset when i did git-review -d just now15:49
fungii'll fix it, thanks for spotting15:49
opendevreviewJeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change
clarkbI've approved ^15:53
clarkb is the gitea 1.20 change and is the resulting held node on the child change16:27
opendevreviewClark Boylan proposed opendev/system-config master: Run bootstrap-bridge with empty nodeset
clarkbinfra-root (and ianw if possible) I think that is a good optimization for our hourly job runs17:02
fungioh, good catch17:05
TheJulialooks like hold 0000000239 is waiting, who would like my ssh pubkey17:10
fungiTheJulia: gimme17:11
* TheJulia sees fungi laughing evilly17:12
TheJuliaerr :)17:12
fungiTheJulia: ssh root@
fungiany time17:16
TheJuliaand in, thanks!17:16
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade Etherpad to 1.9.2
opendevreviewClark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node
clarkbI've put a hold in place for ^ so that we can sanity check things pre upgrade17:52
clarkbzuul today looks like pre feature freeze is happening18:18
fungithere was a big spike in merger requests at 14:34 which i assume corresponded to a bunch of changes getting enqueued all at once. the node request backlog has been burning down for an hour or so at least18:25
fricklerthough that seems to have been submitted starting 14:5518:28
fungithere was a much smaller spike around 14:57 which probably corresponds to those18:35
fungiinfra-root: i've opened ticket #230817-ord-0001055 under our nodepool account in rackspace, requesting they delete the nearly 250 stuck error deleting instances so we can get back to using that third of our overall capacity18:47
fungii'm also doing another manual image upload test to rax-iad, in hopes whatever the problem is with their glance in that region has cleared up18:49
fungiupload completed and the image import task was created in glance at 2023-08-17T18:53:02Z but i'm still looping waiting for it to appear in the image list19:10
clarkbwas the upload any faster than before or was all the time in waiting for it show up in the listing?19:19
clarkbI've got kids gymnastics presentation thing this afternoon I'll pop out early for just a heads up19:19
clarkband probably tomorrow morning I'm going to get out on the bike as our heat wave is ending19:19
fungiupload time was 4m30.807s but i don't think that's necessarily faster/slower than before when i was testing19:21
fungithe delay is pretty much entirely waiting for whatever happens between upload and appearing in the image list19:21
fungibike ride sounds like a great idea. i think we're planning for a nice long walk first thing on saturday morning before it gets unbearably hot19:22
clarkbits been crazy hot since sunday here and even though today is a bit cooler I would've had to get out the door by like 8am for it to be bearable and I'm still too jet lagged for that19:25
clarkbtomorrow morning should be much better19:25
fungithe past month here has been either hot+humid with no breeze, or raining, so i'm way behind on yardwork as a result19:26
fungiappeared sometime between 19:26:48 and 19:27:48 (loop was polling once a minute)19:29
fungiso around 34-35 minutes after upload19:31
fungikeep in mind that's with our uploads to that region paused19:31
fungiso this was just a single upload (from bridge, not even a builder)19:31
npanda_How do I get Genoa support. I am Using Openstack Zed release19:43
npanda_qemu-system-x86_64 -cpu ?|grep -i genoa --> Doesn't return anything19:43
npanda_libvirtd --version returns libvirtd (libvirt) 8.0.019:44
Clark[m]npanda_ this channel is dedicated to discussion of the development tools used to build openstack not openstack itself. I'm not sure what Genoa is or where to get help for that though.19:46
funginpanda_: you may have more luck asking openstack questions in the #openstack channel or on the mailing list19:49
fungi(openDEV is not openSTACK)19:50
fungii'm familiar with the city of genoa, and also their lovely sausages19:53
fungilooks like maybe genoa is the codename for one of amd's cpus19:53
opendevreviewMerged opendev/system-config master: Restart Mailman 3 containers when configs change
fungiyay! now to await the deploy job19:56
fungiwhich, conveniently, is already running19:56
fungiit did update the /var/lib/mailman/web/ file but the uwsgi processes still have a start time from tuesday19:58
fungiRUNNING HANDLER [mailman3 : down containers] skipping: [] => {"changed": false, "skip_reason": "Conditional result was False"}20:00
fungithe "mailman check for running containers" task did get an rc of 0 so it must be the docker-compose stdout that didn't match20:01
funginow i wonder if that was supposed to be on stderr20:01
clarkboh that could be I guess. Maybe test docker-compose up -d on the list server too?20:02
fungiyep, it's on stderr20:03
* fungi sighs20:03
opendevreviewJeremy Stanley proposed opendev/system-config master: Mailman3: check docker-compose stderr not stdout
fungionce more unto the breach!20:06
fungi shows the "deleting" count is rapidly falling over the past few minutes, so my smoke signal may have reached them20:10
clarkbI've approved that if you want to update the file on disk20:10
fungimanually rolled /var/lib/mailman/web/ back to SITE_ID=1 again and new modified time on the file is 20:1120:11
funginevermind abouit the deleting nodes, i was looking at the wrong graph20:12
fungii don't think they've started working on that ticket yet20:12
fungian interesting result of our speculative container builds... the fact that they all rely on a common opendev-buildset-registry job throws a serious wrench into zuul's ability to estimate buildset completion time20:28
fungisince that job will stay paused for varying amounts of time based on whatever the duration of the other jobs relying on it have20:29
fungiso for 891903 it thinks that the builds which need the registry are going to wrap up in another 2-3 minutes, but that the registry job itself needs another half hour after those finish20:30
fungithat said, i can't currently conceive of an elegant estimation algorithm to address that challenge20:31
fungiclarkb: looking at the held gitea upgrade sample, i noticed something that isn't new but is broken... if you go to then at the top there's a "projects" tab which leads to and a 404 page20:36
fungiprobably something we can comment out in the template20:37
fungialso while browsing, the ambiguous unicode character identification on is pretty nifty even if i can't immediately tell which characters on the affected lines are at fault20:41
fungimaybe the apostrophes aren't apostrophes but rather "smart" quotes?20:42
clarkbfungi: I think we can disable repo.projects in the config and that might be what we need to fix the first thing20:45
fungii guess that's where it's coming from?20:46
clarkbya looks like it. So can read projects is true but maybe disabling projects will flip that20:46
clarkbor its a bug and they need to check projects are neabled and readable20:46
clarkbI'm not sure I understand the second thing20:46
clarkboh it annotates the problem lines that helps20:47
fungithe second thing wasn't a bug, i was just trying to figure out what specific characters i considered ambiguous in those lines20:47
fungiand yeah, you have to scroll down to get to them20:47
clarkbah yup if you highlihgt the ' in we'll it tells you that is the problem20:47
clarkbU+2019 confusable with U+006020:47
fungioh, indeed, i didn't notice there was a tooltip20:48
clarkbdon't need to highlight just hover on it with your pointer20:48
clarkbDISABLED_REPO_UNITS = repo.issues,repo.pulls,,repo.projects,repo.actions <- we already set that and I think that is why it is a 40420:48
clarkbso that is a bug that should be fixed upstream imo20:48
fungiright, but it's not clear where to hover since (at least in my browser) there's no marker other that at the beginning of the line. i guess slowly move the mouse cursor along the line until we get a popup tooltip20:49
fungistill pretty cool20:49
clarkbfungi: if you click the alert sign it just barely highlights the characters20:50
fungioh, huh...20:50
clarkbits super missable on my laptop browser20:50
fungiyeah, neat20:50
fungiactually fairly noticeable on my browser since it's assuming dark mode from my defaults so the bright yellow box around it pops nicely20:51
fungii can imagine it would disappear into a bright background easily though20:51
clarkbah with the light mode its just a faint line. I only noticed bceaues the line length changes and it redraws the text area a bit20:51
fungigonna run out to pick up takeout, should be back by the time the mm3 change merges20:52
fungialso the held gitea upgrade sample seems good to me. i didn't find anything broken which wasn't broken already in production anyway20:52
clarkbcool. I think the only other gitea question is how much do we hate confuguring secrets for unused oauth2 features. I've mostly come to terms with it. Fixing that doesn't seem to be a priority upstream and I don't have time to dig into it right now20:53
fungiyeah, i still need to look through the change20:58
opendevreviewMerged opendev/system-config master: Mailman3: check docker-compose stderr not stdout
fungilooks like the deploy finished21:34
funginew /var/lib/mailman/web/ update time is 21:2121:35
fungiuwsgi process start time is 21:2121:35
funginow for the most important part...21:35
fungirefreshed and it finally says "" in the top-left corner!21:36
fungiand with that, i'm going to eat21:37
fungino response on the stuck deleting nodes yet21:37
clarkbI mean for the mailman stuff. We just need patience on the other thing21:41
clarkbfor som ereason I thought the zuul archive sstuff was already done? But I guess not you only did opendev then left things so that zuul could be semi automated in transition?21:41
clarkbsetup up as and hit clarkb-test pad for etherpad 1.9.2 testing. It seems to work for me with multiple browsers21:48
fungiah, no, was the primary site name, and with SITE_ID=1 it appeared on all the archive pages for every domain hosted on that instance22:05
clarkbI see and automation fliped it back to 0 and made things work the wa we want?22:06
fungiwe needed the SITE_ID=0 config change to merge to fix that (turning on domain-specific behavior), which merged yesterday22:06
fungibut the fact that we weren't restarting the containers meant that wasn't getting used by the running processes22:06

Generated by 2.17.3 by Marius Gedminas - find it at!