Thursday, 2022-09-01

*** dviroel is now known as dviroel|out00:09
*** rlandy|bbl is now known as rlandy01:44
*** rlandy is now known as rlandy|out02:20
*** ysandeep|out is now known as ysandeep05:36
opendevreviewIan Wienand proposed opendev/system-config master: Add zuul_console_disable flag to added hosts
*** jpena|off is now known as jpena07:38
*** pojadhav is now known as pojadhav|ruck07:44
mnasiadkaHello - any core willing to merge ?07:54
opendevreviewMerged openstack/project-config master: Add nested-virt-rockylinux-9
mnasiadkathanks frickler 08:14
mnasiadkafrickler: any idea when the label will show up?08:27
fricklermnasiadka: looks like it is there already?08:43
mnasiadkafrickler: ok, I was just too fast ;)08:48
fricklermnasiadka: the deploy needed to wait for our hourly periodic jobs to finish first I think
fricklerthat reminds me I still wanted to add my gpg key for the log encryption08:52
mnasiadkaSeems there's no EPEL repository installed on rockylinux-9 nodes?10:21
*** rlandy|out is now known as rlandy10:31
*** dviroel|out is now known as dviroel11:27
*** ysandeep is now known as ysandeep|afk11:51
opendevreviewyatin proposed opendev/system-config master: Revert "Use rackspace mirror to sync centos stream repos"
ykarelclarkb, fungi can you check ^12:11
fungiykarel: can the centos community suggest a more stable mirror? we seem to keep switching back and forth every few weeks12:12
ykarelfungi, iirc when we checked in past they said they don't control those mirrors12:13
fungiright, is there one they do control which is for public use and supports rsync?12:13
ykarelbut can recheck and ask again12:14
fungibut also, if our tolerance is really that mirrors need a <24 hour freshness, that seems like a pretty high bar12:14
ykarelfungi, the issue is happening as the new nodepool images are built with latest c9 packages12:16
ykarelseems those images builds daily?12:16
ykarelbut then mirrors are not up to date and have old packages causing these issues12:16
fungiwe avoid this for ubuntu images by building from our mirrors. did we somehow not do that for centos image builds?12:16
ykarelme not sure, likely ^ is issue with centos12:17
ykarelbut worth to check/confirm12:17
fungiunfortunately just skimming i can't tell where it's getting the packages12:19
fungilooks like we control it by setting a DIB_DISTRIBUTION_MIRROR envvar for some images in here:
ykarelfungi, ^ this latest image logs?12:23
ykarelit's from 2022-08-3012:23
ykarelbut we started seeing issues today12:23
fungiykarel: not necessarily, i picked one of the builders at random but nb02 is probably the one that built today's image12:23
ykarelack /me checks there12:24
ykarelyes that's latest12:24
ykarelNetworkManager-1.40.0-1.el9.x86_64.rpm: Status code: 404 for
ykarelseems non mirrors repos are also enabled there12:25
fungilike maybe dnf has a fallback to another mirror?12:27
ykarelfrom logs i see centos-stream-repos get's installed, which contains default repos12:31
fungiyeah, a git grep DIB_DISTRIBUTION_MIRROR in the dib repo suggests that the yum-minimal element (which centos-minimal depends on) doesn't "yet" support DIB_DISTRIBUTION_MIRROR12:32
fungiaccording to a todo comment in there12:32
fungiso that's probably why we're not bothering to set that envvar in our nodepool config for the centos and fedora images12:33
ykarelyes seems so12:34
*** ysandeep|afk is now known as ysandeep12:38
ykarelbut seeing references of in logs , means it's set somewhere12:39
fungiespecially interesting given that's not one of our mirrors12:40
fungiours would be like mirror.dfw.rax.opendev.org12:41
funginotice it tries their dfw mirror and then their ord mirror12:42
fungimaybe that's dnf smartly picking the nearest official mirror?12:42
ykarelyes it does that12:43
ykareli see
ykarelso will best mirror from
fungiso that's "expected" behavior i guess, it's not something we're setting12:44
fungi(as far as the fact that it's picking the rackspace mirrors, it's because dnf has figured out the builder is in rackspace's network)12:44
fungiand yes, i agree we probably need to make the base.repo file template that metalink field or have some separate element replace that file with a different one before it gets used12:45
ykarelyes right12:46
ykareland also iirc there were some issue when using metalink= as rpms are distributed across repos baseos vs appstream12:47
ykareland it can choose different mirror for these repos and cause issues, not sure if that issue is fixed12:48
fungiso anyway, if you or someone you know is interested in adding DIB_DISTRIBUTION_MIRROR support to the centos-minimal element, that would probably go a long way to mitigating these sorts of issues12:48
fungiopeneuler-minimal is rpm-based and seems to support it, so might be able to copy some bits from there12:49
ykarelfungi, yes agree that will help in avoiding such issues12:50
ykarelbut for now i think can switch to facebook mirror12:50
ykarelwill ask tripleo ci folks to get it fixed and avoid this in future12:50
fungiyeah, looks like zuul came back with a thumbs-up a few minutes ago12:50
ykarelchandankumar, rlandy jm1 can you take it into your radar ^12:51
fungii'll upgrade my +2 to a +3 so this doesn't get dragged out even longer12:51
ykarelThanks fungi 12:51
rlandywill read back in a bit12:53
fungimnasiadka: maybe it just needs the epel element added to its elements list like rl8 has?
fungilooks like they both set an envvar to make epel disabled by default, but only rl8 includes the element to add epel at all12:54
*** dasm|off is now known as dasm12:54
fungimnasiadka: is the change which added 9 to the config, and did so with no epel even though the 8 entry directly above it had epel already at that time. no mention in the commit message nor review comments of why it might have been omitted, so perhaps it was simply overlooked?12:58
fungiNeilHanlon: ^ maybe you know the reason?12:59
NeilHanlonfungi: oversight. I can put in a change to add it back in. TY13:04
fungiNeilHanlon: no worries, thanks for confirming!13:05
ykarelThu  1 Sep 10:00:02 UTC 202213:05
ykarelrackspace mirror got updated13:06
ykarelfungi, ^ i think it's fine to abandon for now13:06
opendevreviewNeil Hanlon proposed openstack/project-config master: Add epel element to Rocky Linux 9 configuration
mnasiadkafungi, NeilHanlon - thanks, was just asking! :)13:07
ykarelfungi, ok ignore seems still ok to switch to facebook one13:08
ykarelwe need to seen what's the frequency of those mirrors sync, iirc it was 6 hours for facebook one13:09
fungiykarel: yeah, also our cronjob is set to run every 4 hours at 7 minutes after the hour, so our next pull should start at 16:07 utc13:10
ykarelso approx 3 hours from now13:15
NeilHanlonfungi: here's a stupid question for you... does the DIB_DISABLE_EPEL flag just disable EPEL __during__ the DIB process?13:16
opendevreviewMerged opendev/system-config master: Revert "Use rackspace mirror to sync centos stream repos"
fungiNeilHanlon: it makes it so that you have to tell the package manager that you want to use epel, so that jobs don't accidentally install packages from epel without turning it on13:23
fungioh, actually i think you're right, it may only be disabled during image building. let me find the docs13:24
Clark[m]fungi: ykarel: it auto picking the rax mirror due to locality is probably a good reason to keep syncing from there in the meantime.13:25
Clark[m]fungi: NeilHanlon I believe it sets a disabled flag in the definition file which ya forces install commands to explicitly enable it13:26
fungiNeilHanlon: no, my memory was right. see
fungi"To disable the EPEL repo (but leave it available if used with an explicit --enablerepo) set this to 1"13:26
fungiokay, returning to the meetpad config refresh from yesterday again, i think the jitsi-meet_7648 tag in the jitsi-meet repo is what corresponds to the stable-7648-4 tag from the docker-jitsi-meet repo, so hopefully if i use those that's what will reflect the defaults in the image13:29
NeilHanlonfungi, Clark[m], thank you, makes sense!13:47
fungidigging in the jitsi-meet repo, looks like useRoomAsSharedDocumentName was removed in 1d44741. trying to figure out if that became the default behavior or something13:58
fungii find references to it in jicofo's history, as well as still in the participant script in jxs14:01
fungiokay, so
fungiand then later seems to have dropped that option and introduced a getUseRandomSharedDocumentName which inverted the behavior?14:07
clarkbfungi: those changes are old enough that we would've noticed in our install if the old configs were no logner supported I think14:08
clarkbit almost seems like using the room as the shared doc name is the default and you have to set useRandomSharedDocumentName to get that behavior. Which I guess would explain why we continued to function (the new default is what we had explicitly set before)14:09
fungiyes, i think what's going on is that our "config.useRoomAsSharedDocumentName = true;" is being ignored since it 1. moved to jicofo, 2. got replaced by a different config option with inverted semantics, and so 3. is now the default behavior anyway14:10
fungii'll just drop it from the new config and note that in the commit message14:10
clarkblooks like the issue with the mm3 change may have been the client specifier on max_allowed_packet. Changing that to mysqldump appears to have fixed the deployment.14:16
opendevreviewMerged openstack/project-config master: Add epel element to Rocky Linux 9 configuration
*** ysandeep is now known as ysandeep|afk14:22
*** pojadhav|ruck is now known as pojadhav|afk14:23
fungiclarkb: when updating the docker-compose files in our jitsi-meet role to track the stable tag, what should i sync from the upstream docker-jitsi-meet repo? should i update the environment list? i see they also added a default restart policy, and tacked :Z onto the ends of the volume maps14:24
fungiand the version string is updated to 3.514:25
clarkbfungi: :Z appears selinux specific so won't have the desired effect for us. Not sure if docker just ignores it14:26
clarkbthe intended effect seems to be that the bind mount isn't shared between containers14:27
fungiis the "version: '2'" in our compose files related to the version of docker-compose we're using, or is it safe to update that?14:28
clarkbFor the environment list anything yuo want passed from the .env files has to be listed in the environment list otherwise it won't actually be passed to the container iirc14:28
clarkbdefault restart policy is probably useful.14:28
clarkbThe version is the version of the configuration14:28
fungiokay, so if they added new envvars in the env.example then they presumably updated the environment lists in the compose file in tandem with that, and we should update as well14:29
clarkbOur docker-compose installation needs to be new enough to support it. I think we install a fairly new docker compose14:29
clarkbThe main thing is if you add any features that actually need new docker compose config14:29
clarkbif you don't I would keep the old version set as it is more flexible14:29
fungii'll just check the ones we're setting against the list we have, in that case14:30
fungishould i remove any we don't set, for clarity?14:30
clarkbI think that was part of what we were trying to do. We only awnted to apss through the subset of env vars that made sense14:31
fungias for the image specification, theirs look like jitsi/jvb:${JITSI_IMAGE_VERSION:-stable-7648-4} but i guess we just want instead?14:31
fungicool, thanks!14:32
fungii see envvars we're setting in our jvb-env.j2 which aren't in the environment list in our jvb-docker-compose.yaml, is that expected?14:35
fungioh, i see some of those seem copied from the other env file. maybe i should clean that up14:36
clarkbjvb is a subset14:37
opendevreviewWill Szumski proposed openstack/diskimage-builder master: Adds rocky-container-generic
opendevreviewJeremy Stanley proposed opendev/system-config master: Update Jitsi configs to latest upstream samples
*** pojadhav|afk is now known as pojadhav|ruck15:06
*** ysandeep|afk is now known as ysandeep15:21
opendevreviewWill Szumski proposed openstack/diskimage-builder master: Adds passwd to rocky-container os packages
clarkbfungi: I guess we decided we needed the newer docker compose config version? maybe for the restart: unless stopped?15:31
fungiclarkb: i just did what i could to try and keep it in sync with the upstream compose file15:32
fungiso we have less divergence the next time we try to compare15:33
clarkbfungi: ok. I would double check the docker-compose version on the meetpad servers and ensure it supports that config file version. But if it does then it should be fine15:33
clarkbMy meeting is done. Any opinion on whether or not it would be helpful to jointhe openstack tc call now? Otherwise breakfast15:34
fungino clue, i'm chairing the security sig meeting still, so didn't try to dial in15:34
clarkbya I had a call so couldn't dial in at the start time either. I'll go get breakfast then15:36
clarkbBut then I'll look at the jitsi change more closely and take my day from there15:36
*** ysandeep is now known as ysandeep|out15:40
clarkbThe "make json parser happy" line at the end of the interface config js file is a funny one15:46
clarkbI wonder how many bugs had been filed over that until they added the line15:46
fungiclarkb: the scant meeting notes in #openstack-tc indicate there's a desire to sync up with the i18n sig and it was suggested that there would indeed be an infrastructure component to that discussion15:49
fungiat the ptg, specifically15:49
*** dviroel is now known as dviroel|lunch15:59
clarkbfungi: comments posted. The most interesting ones are probably to do with the jvb env file template16:41
clarkbI feel like this change is safe and should be fine. But we should definitely plan to test meetpad after we update16:45
johnsomFYI, the "parent" link in gerrit doesn't seem to work anymore. If you click on it from this patch for example: it comes up with no changes now.16:49
*** jpena is now known as jpena|off16:50
clarkbjohnsom: I think that is beacuse the parent is a merge commit and gerrit doesn't have a change for that merge commit16:53
fungijohnsom: i don't see where it says "parent" anywhere there16:53
clarkbfungi: you have to click the show all button furst16:54
fungithough i'll admit i quickly get lost in that webui16:54
clarkbbut its doing a query for the merge commit sha in the change list and no such change exists16:54
johnsomYeah, that changed too a while ago, you have to "SHOW ALL" to see the parent link now16:54
fungioh, found. thanks!16:54
johnsomYeah, I noticed it didn't work, rebased the patch, and it still didn't work. This used to work correctly and is very handy to find the parent patch. Without it I have to clone local to see what it's based on.16:55
fungiyeah, it looks like that gets linked to a gerrit change search for a commit id, but if the commit id doesn't correspond to a change gerrit returns nothing16:56
fungijohnsom: what did it return in the past when the parent wasn't a gerrit change16:56
fungidid it link to a git repository browser instead?16:56
johnsomIt always just took me to the parent patch in gerrit16:57
fungii mean when the parent isn't a gerrit patch16:57
fungilike in this case16:57
johnsomIn my experience it has never not gone to the proper parent patch in gerrit16:57
clarkbyou've either always gotten lucky clicking on parents that re changes or it did something like link to gitweb16:58
johnsomI don't know if the "REBASE" button changed behavior or if the parent link did.16:58
clarkbit wouldn't surprise me if you usually click on it in a stack16:58
clarkbI don't know that either have. Just that in specific cases it can't be fulfiiled16:59
clarkbit depends on the input data16:59
johnsomI use it pretty regularly on some of these evil patch chains. So, I'm personally not convinced it was luck.17:00
clarkbyes evil patch chains are where it should work si what I'm saying17:00
fungijohnsom: try the parent link on this for comparison:
clarkbbeacuse you typically don't include merge commits in those chains and every commit in the chain has a corresponding change17:00
fungiand yeah, if the change was a fast-forward instead of needing a merge commit (even if it wasn't part of an explicit change series in gerrit) then the parent link will take you to an actual change17:02
fungier, was a fast-forward from an actual change commit17:02
johnsomYeah, that works as expected. So what you are saying is it will always be broken when the patch is based off master, but not when based on a open patch.17:02
funginot necessarily17:02
clarkbsort of. It will always be broken when the parent is a merge commit17:02
fungionly if the patch is based off a merge commit (which the branch tip often is)17:02
clarkbsometimes master isnt' a merge commit and it would work then too17:02
fungiyou can still plug that git commit id into gitea though to find the merge commit, and its second parent will be whatever change got merged to create that merge commit17:04
johnsomOk, I get your point. Seems odd, but I get it. Thanks for the explanation. 17:04
*** dviroel|lunch is now known as dviroel17:18
fungiclarkb: i left a followup question on 855388 in response to your question17:29
clarkbfungi: responded thanks17:38
opendevreviewJeremy Stanley proposed opendev/system-config master: Update Jitsi configs to latest upstream samples
fungiclarkb: and to be clear, i wasn't setting them empty but rather leaving them empty like the upstream version:
fungibut commenting them out like i needed to in the other env makes our two env files more consistent with each other, yes17:47
clarkboh I see. I guess they leave it uncommented to impress upon you that you need to set them. But those aren't valid for the jvb only server iirc17:51
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
fungipython packaging survey is up:
fungipeople who have opinions about python packaging topics may want to supply them there20:42
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks
*** dviroel is now known as dviroel|afk20:57
*** dasm is now known as dasm|off21:42
clarkbianw: if you get a chance can you review fungi's jitsi meet config update change? I think we'll be able to test that tomorrow if we can have someone else sanity check it22:24
clarkbIn theory its a complete noop but a lot of moving pieces so good to have eyeballs22:24
ianwwill do22:24
fungiyeah, it's really just updating our copied configs to match current stable upstream versions, and switching to their "stable" container image tag (which ironically is more recent than their "latest" tag)22:25
fungidriven by the fact that turning on the pre-join page feature seems to stop firefox from thinking the audio stream is unsolicited, and that setting became an upstream default a year or more ago22:26
fungiwe've just been broken because we were frozen on increasingly ancient configuration22:27
clarkbfungi: also maybe tomorrow too?22:58
fungioh, yeah, for some reason i thought i'd already reviewed that one23:01
clarkbI'm just looking at my backlog and making sure nothing is missed23:01
fungiin that case, 852584 and topic:no-rebase could use a quick look from someone23:02
fungilooks like you already reviewed most of those23:03
fungii meant to bring up for broader discussion but i think i forgot to add it to the meeting agenda or raise it on the ml23:04
fungii guess that's why it's still wip23:04
*** dviroel|afk is now known as dviroel23:55

Generated by 2.17.3 by Marius Gedminas - find it at!