*** dviroel is now known as dviroel|out | 00:09 | |
*** rlandy|bbl is now known as rlandy | 01:44 | |
*** rlandy is now known as rlandy|out | 02:20 | |
*** ysandeep|out is now known as ysandeep | 05:36 | |
opendevreview | Ian Wienand proposed opendev/system-config master: Add zuul_console_disable flag to added hosts https://review.opendev.org/c/opendev/system-config/+/855472 | 06:14 |
---|---|---|
*** jpena|off is now known as jpena | 07:38 | |
*** pojadhav is now known as pojadhav|ruck | 07:44 | |
mnasiadka | Hello - any core willing to merge https://review.opendev.org/c/openstack/project-config/+/854431 ? | 07:54 |
opendevreview | Merged openstack/project-config master: Add nested-virt-rockylinux-9 https://review.opendev.org/c/openstack/project-config/+/854431 | 08:08 |
mnasiadka | thanks frickler | 08:14 |
mnasiadka | frickler: any idea when the label will show up? | 08:27 |
frickler | mnasiadka: looks like it is there already? | 08:43 |
mnasiadka | frickler: ok, I was just too fast ;) | 08:48 |
frickler | mnasiadka: the deploy needed to wait for our hourly periodic jobs to finish first I think https://zuul.opendev.org/t/openstack/build/efb5a5ccc8c74edc8b4efc47563065a5 | 08:50 |
frickler | that reminds me I still wanted to add my gpg key for the log encryption | 08:52 |
mnasiadka | Seems there's no EPEL repository installed on rockylinux-9 nodes? | 10:21 |
*** rlandy|out is now known as rlandy | 10:31 | |
*** dviroel|out is now known as dviroel | 11:27 | |
*** ysandeep is now known as ysandeep|afk | 11:51 | |
opendevreview | yatin proposed opendev/system-config master: Revert "Use rackspace mirror to sync centos stream repos" https://review.opendev.org/c/opendev/system-config/+/855457 | 12:09 |
ykarel | clarkb, fungi can you check ^ | 12:11 |
fungi | ykarel: can the centos community suggest a more stable mirror? we seem to keep switching back and forth every few weeks | 12:12 |
ykarel | fungi, iirc when we checked in past they said they don't control those mirrors | 12:13 |
fungi | right, is there one they do control which is for public use and supports rsync? | 12:13 |
ykarel | but can recheck and ask again | 12:14 |
fungi | but also, if our tolerance is really that mirrors need a <24 hour freshness, that seems like a pretty high bar | 12:14 |
ykarel | fungi, the issue is happening as the new nodepool images are built with latest c9 packages | 12:16 |
ykarel | seems those images builds daily? | 12:16 |
ykarel | but then mirrors are not up to date and have old packages causing these issues | 12:16 |
fungi | we avoid this for ubuntu images by building from our mirrors. did we somehow not do that for centos image builds? | 12:16 |
ykarel | me not sure, likely ^ is issue with centos | 12:17 |
ykarel | but worth to check/confirm | 12:17 |
fungi | unfortunately just skimming https://nb01.opendev.org/centos-9-stream-0000007720.log i can't tell where it's getting the packages | 12:19 |
fungi | looks like we control it by setting a DIB_DISTRIBUTION_MIRROR envvar for some images in here: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml | 12:22 |
ykarel | fungi, ^ this latest image logs? | 12:23 |
ykarel | it's from 2022-08-30 | 12:23 |
ykarel | but we started seeing issues today | 12:23 |
fungi | ykarel: not necessarily, i picked one of the builders at random but nb02 is probably the one that built today's image | 12:23 |
ykarel | ack /me checks there | 12:24 |
ykarel | yes that's latest | 12:24 |
ykarel | https://nb02.opendev.org/centos-9-stream-0000007722.log | 12:24 |
ykarel | NetworkManager-1.40.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/NetworkManager-1.40.0-1.el9.x86_64.rpm | 12:25 |
ykarel | seems non mirrors repos are also enabled there | 12:25 |
fungi | like maybe dnf has a fallback to another mirror? | 12:27 |
ykarel | from logs i see centos-stream-repos get's installed, which contains default repos | 12:31 |
fungi | yeah, a git grep DIB_DISTRIBUTION_MIRROR in the dib repo suggests that the yum-minimal element (which centos-minimal depends on) doesn't "yet" support DIB_DISTRIBUTION_MIRROR | 12:32 |
fungi | according to a todo comment in there | 12:32 |
fungi | so that's probably why we're not bothering to set that envvar in our nodepool config for the centos and fedora images | 12:33 |
ykarel | yes seems so | 12:34 |
*** ysandeep|afk is now known as ysandeep | 12:38 | |
ykarel | but seeing references of https://dfw.mirror.rackspace.com in logs , means it's set somewhere | 12:39 |
fungi | especially interesting given that's not one of our mirrors | 12:40 |
fungi | ours would be like mirror.dfw.rax.opendev.org | 12:41 |
fungi | notice it tries their dfw mirror and then their ord mirror | 12:42 |
fungi | maybe that's dnf smartly picking the nearest official mirror? | 12:42 |
ykarel | yes it does that | 12:43 |
ykarel | i see https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/centos-minimal/yum.repos.d/9-stream/base.repo | 12:43 |
ykarel | so will best mirror from https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64 | 12:44 |
fungi | so that's "expected" behavior i guess, it's not something we're setting | 12:44 |
fungi | (as far as the fact that it's picking the rackspace mirrors, it's because dnf has figured out the builder is in rackspace's network) | 12:44 |
fungi | and yes, i agree we probably need to make the base.repo file template that metalink field or have some separate element replace that file with a different one before it gets used | 12:45 |
ykarel | yes right | 12:46 |
ykarel | and also iirc there were some issue when using metalink= as rpms are distributed across repos baseos vs appstream | 12:47 |
ykarel | and it can choose different mirror for these repos and cause issues, not sure if that issue is fixed | 12:48 |
fungi | so anyway, if you or someone you know is interested in adding DIB_DISTRIBUTION_MIRROR support to the centos-minimal element, that would probably go a long way to mitigating these sorts of issues | 12:48 |
fungi | openeuler-minimal is rpm-based and seems to support it, so might be able to copy some bits from there | 12:49 |
ykarel | fungi, yes agree that will help in avoiding such issues | 12:50 |
ykarel | but for now i think can switch to facebook mirror | 12:50 |
ykarel | will ask tripleo ci folks to get it fixed and avoid this in future | 12:50 |
fungi | yeah, looks like zuul came back with a thumbs-up a few minutes ago | 12:50 |
ykarel | chandankumar, rlandy jm1 can you take it into your radar ^ | 12:51 |
fungi | i'll upgrade my +2 to a +3 so this doesn't get dragged out even longer | 12:51 |
ykarel | Thanks fungi | 12:51 |
fungi | yw | 12:52 |
rlandy | will read back in a bit | 12:53 |
fungi | mnasiadka: maybe it just needs the epel element added to its elements list like rl8 has? https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L174-L195 | 12:53 |
fungi | looks like they both set an envvar to make epel disabled by default, but only rl8 includes the element to add epel at all | 12:54 |
*** dasm|off is now known as dasm | 12:54 | |
fungi | mnasiadka: https://review.opendev.org/852167 is the change which added 9 to the config, and did so with no epel even though the 8 entry directly above it had epel already at that time. no mention in the commit message nor review comments of why it might have been omitted, so perhaps it was simply overlooked? | 12:58 |
fungi | NeilHanlon: ^ maybe you know the reason? | 12:59 |
NeilHanlon | fungi: oversight. I can put in a change to add it back in. TY | 13:04 |
fungi | NeilHanlon: no worries, thanks for confirming! | 13:05 |
ykarel | Thu 1 Sep 10:00:02 UTC 2022 | 13:05 |
ykarel | rackspace mirror got updated | 13:06 |
ykarel | fungi, ^ i think it's fine to abandon for now | 13:06 |
opendevreview | Neil Hanlon proposed openstack/project-config master: Add epel element to Rocky Linux 9 configuration https://review.opendev.org/c/openstack/project-config/+/855509 | 13:06 |
mnasiadka | fungi, NeilHanlon - thanks, was just asking! :) | 13:07 |
NeilHanlon | :) | 13:07 |
ykarel | fungi, ok ignore seems still ok to switch to facebook one | 13:08 |
ykarel | we need to seen what's the frequency of those mirrors sync, iirc it was 6 hours for facebook one | 13:09 |
fungi | ykarel: yeah, also our cronjob is set to run every 4 hours at 7 minutes after the hour, so our next pull should start at 16:07 utc | 13:10 |
ykarel | so approx 3 hours from now | 13:15 |
NeilHanlon | fungi: here's a stupid question for you... does the DIB_DISABLE_EPEL flag just disable EPEL __during__ the DIB process? | 13:16 |
opendevreview | Merged opendev/system-config master: Revert "Use rackspace mirror to sync centos stream repos" https://review.opendev.org/c/opendev/system-config/+/855457 | 13:17 |
fungi | NeilHanlon: it makes it so that you have to tell the package manager that you want to use epel, so that jobs don't accidentally install packages from epel without turning it on | 13:23 |
fungi | oh, actually i think you're right, it may only be disabled during image building. let me find the docs | 13:24 |
Clark[m] | fungi: ykarel: it auto picking the rax mirror due to locality is probably a good reason to keep syncing from there in the meantime. | 13:25 |
Clark[m] | fungi: NeilHanlon I believe it sets a disabled flag in the definition file which ya forces install commands to explicitly enable it | 13:26 |
fungi | NeilHanlon: no, my memory was right. see https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/epel/README.rst | 13:26 |
fungi | "To disable the EPEL repo (but leave it available if used with an explicit --enablerepo) set this to 1" | 13:26 |
fungi | okay, returning to the meetpad config refresh from yesterday again, i think the jitsi-meet_7648 tag in the jitsi-meet repo is what corresponds to the stable-7648-4 tag from the docker-jitsi-meet repo, so hopefully if i use those that's what will reflect the defaults in the image | 13:29 |
NeilHanlon | fungi, Clark[m], thank you, makes sense! | 13:47 |
fungi | digging in the jitsi-meet repo, looks like useRoomAsSharedDocumentName was removed in 1d44741. trying to figure out if that became the default behavior or something | 13:58 |
fungi | i find references to it in jicofo's history, as well as still in the participant script in jxs | 14:01 |
fungi | okay, so https://github.com/jitsi/lib-jitsi-meet/pull/1424 | 14:02 |
fungi | and then later https://github.com/jitsi/jicofo/pull/644 seems to have dropped that option and introduced a getUseRandomSharedDocumentName which inverted the behavior? | 14:07 |
clarkb | fungi: those changes are old enough that we would've noticed in our install if the old configs were no logner supported I think | 14:08 |
clarkb | it almost seems like using the room as the shared doc name is the default and you have to set useRandomSharedDocumentName to get that behavior. Which I guess would explain why we continued to function (the new default is what we had explicitly set before) | 14:09 |
fungi | yes, i think what's going on is that our "config.useRoomAsSharedDocumentName = true;" is being ignored since it 1. moved to jicofo, 2. got replaced by a different config option with inverted semantics, and so 3. is now the default behavior anyway | 14:10 |
fungi | i'll just drop it from the new config and note that in the commit message | 14:10 |
clarkb | looks like the issue with the mm3 change may have been the client specifier on max_allowed_packet. Changing that to mysqldump appears to have fixed the deployment. | 14:16 |
opendevreview | Merged openstack/project-config master: Add epel element to Rocky Linux 9 configuration https://review.opendev.org/c/openstack/project-config/+/855509 | 14:16 |
*** ysandeep is now known as ysandeep|afk | 14:22 | |
*** pojadhav|ruck is now known as pojadhav|afk | 14:23 | |
fungi | clarkb: when updating the docker-compose files in our jitsi-meet role to track the stable tag, what should i sync from the upstream docker-jitsi-meet repo? should i update the environment list? i see they also added a default restart policy, and tacked :Z onto the ends of the volume maps | 14:24 |
fungi | and the version string is updated to 3.5 | 14:25 |
clarkb | fungi: :Z appears selinux specific so won't have the desired effect for us. Not sure if docker just ignores it | 14:26 |
clarkb | the intended effect seems to be that the bind mount isn't shared between containers | 14:27 |
fungi | is the "version: '2'" in our compose files related to the version of docker-compose we're using, or is it safe to update that? | 14:28 |
clarkb | For the environment list anything yuo want passed from the .env files has to be listed in the environment list otherwise it won't actually be passed to the container iirc | 14:28 |
clarkb | default restart policy is probably useful. | 14:28 |
clarkb | The version is the version of the configuration | 14:28 |
fungi | okay, so if they added new envvars in the env.example then they presumably updated the environment lists in the compose file in tandem with that, and we should update as well | 14:29 |
clarkb | Our docker-compose installation needs to be new enough to support it. I think we install a fairly new docker compose | 14:29 |
clarkb | The main thing is if you add any features that actually need new docker compose config | 14:29 |
clarkb | if you don't I would keep the old version set as it is more flexible | 14:29 |
fungi | i'll just check the ones we're setting against the list we have, in that case | 14:30 |
fungi | should i remove any we don't set, for clarity? | 14:30 |
clarkb | I think that was part of what we were trying to do. We only awnted to apss through the subset of env vars that made sense | 14:31 |
fungi | as for the image specification, theirs look like jitsi/jvb:${JITSI_IMAGE_VERSION:-stable-7648-4} but i guess we just want docker.io/jitsi/jvb:stable instead? | 14:31 |
clarkb | ya | 14:32 |
fungi | cool, thanks! | 14:32 |
fungi | i see envvars we're setting in our jvb-env.j2 which aren't in the environment list in our jvb-docker-compose.yaml, is that expected? | 14:35 |
fungi | oh, i see some of those seem copied from the other env file. maybe i should clean that up | 14:36 |
clarkb | jvb is a subset | 14:37 |
fungi | yep | 14:40 |
opendevreview | Will Szumski proposed openstack/diskimage-builder master: Adds rocky-container-generic https://review.opendev.org/c/openstack/diskimage-builder/+/855521 | 14:44 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Update Jitsi configs to latest upstream samples https://review.opendev.org/c/opendev/system-config/+/855388 | 14:51 |
*** pojadhav|afk is now known as pojadhav|ruck | 15:06 | |
*** ysandeep|afk is now known as ysandeep | 15:21 | |
opendevreview | Will Szumski proposed openstack/diskimage-builder master: Adds passwd to rocky-container os packages https://review.opendev.org/c/openstack/diskimage-builder/+/855530 | 15:30 |
clarkb | fungi: I guess we decided we needed the newer docker compose config version? maybe for the restart: unless stopped? | 15:31 |
fungi | clarkb: i just did what i could to try and keep it in sync with the upstream compose file | 15:32 |
fungi | so we have less divergence the next time we try to compare | 15:33 |
clarkb | fungi: ok. I would double check the docker-compose version on the meetpad servers and ensure it supports that config file version. But if it does then it should be fine | 15:33 |
clarkb | My meeting is done. Any opinion on whether or not it would be helpful to jointhe openstack tc call now? Otherwise breakfast | 15:34 |
fungi | no clue, i'm chairing the security sig meeting still, so didn't try to dial in | 15:34 |
clarkb | ya I had a call so couldn't dial in at the start time either. I'll go get breakfast then | 15:36 |
clarkb | But then I'll look at the jitsi change more closely and take my day from there | 15:36 |
fungi | thanks! | 15:36 |
*** ysandeep is now known as ysandeep|out | 15:40 | |
clarkb | The "make json parser happy" line at the end of the interface config js file is a funny one | 15:46 |
clarkb | I wonder how many bugs had been filed over that until they added the line | 15:46 |
fungi | clarkb: the scant meeting notes in #openstack-tc indicate there's a desire to sync up with the i18n sig and it was suggested that there would indeed be an infrastructure component to that discussion | 15:49 |
fungi | at the ptg, specifically | 15:49 |
*** dviroel is now known as dviroel|lunch | 15:59 | |
clarkb | fungi: comments posted. The most interesting ones are probably to do with the jvb env file template | 16:41 |
clarkb | I feel like this change is safe and should be fine. But we should definitely plan to test meetpad after we update | 16:45 |
fungi | absolutely | 16:49 |
johnsom | FYI, the "parent" link in gerrit doesn't seem to work anymore. If you click on it from this patch for example: https://review.opendev.org/c/openstack/octavia/+/656811 it comes up with no changes now. | 16:49 |
*** jpena is now known as jpena|off | 16:50 | |
clarkb | johnsom: I think that is beacuse the parent is a merge commit and gerrit doesn't have a change for that merge commit | 16:53 |
fungi | johnsom: i don't see where it says "parent" anywhere there | 16:53 |
clarkb | fungi: you have to click the show all button furst | 16:54 |
fungi | though i'll admit i quickly get lost in that webui | 16:54 |
clarkb | but its doing a query for the merge commit sha in the change list and no such change exists | 16:54 |
johnsom | Yeah, that changed too a while ago, you have to "SHOW ALL" to see the parent link now | 16:54 |
fungi | oh, found. thanks! | 16:54 |
johnsom | Yeah, I noticed it didn't work, rebased the patch, and it still didn't work. This used to work correctly and is very handy to find the parent patch. Without it I have to clone local to see what it's based on. | 16:55 |
fungi | yeah, it looks like that gets linked to a gerrit change search for a commit id, but if the commit id doesn't correspond to a change gerrit returns nothing | 16:56 |
fungi | johnsom: what did it return in the past when the parent wasn't a gerrit change | 16:56 |
fungi | ? | 16:56 |
fungi | did it link to a git repository browser instead? | 16:56 |
johnsom | It always just took me to the parent patch in gerrit | 16:57 |
fungi | i mean when the parent isn't a gerrit patch | 16:57 |
fungi | like in this case | 16:57 |
johnsom | In my experience it has never not gone to the proper parent patch in gerrit | 16:57 |
clarkb | you've either always gotten lucky clicking on parents that re changes or it did something like link to gitweb | 16:58 |
johnsom | I don't know if the "REBASE" button changed behavior or if the parent link did. | 16:58 |
clarkb | it wouldn't surprise me if you usually click on it in a stack | 16:58 |
clarkb | I don't know that either have. Just that in specific cases it can't be fulfiiled | 16:59 |
clarkb | it depends on the input data | 16:59 |
johnsom | I use it pretty regularly on some of these evil patch chains. So, I'm personally not convinced it was luck. | 17:00 |
clarkb | yes evil patch chains are where it should work si what I'm saying | 17:00 |
fungi | johnsom: try the parent link on this for comparison: https://review.opendev.org/c/openstack/octavia/+/849129/30 | 17:00 |
clarkb | beacuse you typically don't include merge commits in those chains and every commit in the chain has a corresponding change | 17:00 |
fungi | and yeah, if the change was a fast-forward instead of needing a merge commit (even if it wasn't part of an explicit change series in gerrit) then the parent link will take you to an actual change | 17:02 |
fungi | er, was a fast-forward from an actual change commit | 17:02 |
johnsom | Yeah, that works as expected. So what you are saying is it will always be broken when the patch is based off master, but not when based on a open patch. | 17:02 |
fungi | not necessarily | 17:02 |
clarkb | sort of. It will always be broken when the parent is a merge commit | 17:02 |
fungi | only if the patch is based off a merge commit (which the branch tip often is) | 17:02 |
clarkb | sometimes master isnt' a merge commit and it would work then too | 17:02 |
fungi | you can still plug that git commit id into gitea though to find the merge commit, and its second parent will be whatever change got merged to create that merge commit | 17:04 |
johnsom | Ok, I get your point. Seems odd, but I get it. Thanks for the explanation. | 17:04 |
*** dviroel|lunch is now known as dviroel | 17:18 | |
fungi | clarkb: i left a followup question on 855388 in response to your question | 17:29 |
clarkb | fungi: responded thanks | 17:38 |
fungi | done | 17:41 |
opendevreview | Jeremy Stanley proposed opendev/system-config master: Update Jitsi configs to latest upstream samples https://review.opendev.org/c/opendev/system-config/+/855388 | 17:41 |
fungi | thanks! | 17:41 |
fungi | clarkb: and to be clear, i wasn't setting them empty but rather leaving them empty like the upstream version: https://github.com/jitsi/docker-jitsi-meet/blob/master/env.example#L191-L204 | 17:47 |
fungi | but commenting them out like i needed to in the other env makes our two env files more consistent with each other, yes | 17:47 |
clarkb | oh I see. I guess they leave it uncommented to impress upon you that you need to set them. But those aren't valid for the jvb only server iirc | 17:51 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 18:03 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 18:12 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 18:25 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 18:45 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 19:39 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 20:23 |
fungi | python packaging survey is up: https://www.surveymonkey.co.uk/r/6TLZH3P | 20:41 |
fungi | people who have opinions about python packaging topics may want to supply them there | 20:42 |
opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Speed up log file fetching tasks https://review.opendev.org/c/zuul/zuul-jobs/+/855402 | 20:43 |
*** dviroel is now known as dviroel|afk | 20:57 | |
*** dasm is now known as dasm|off | 21:42 | |
clarkb | ianw: if you get a chance can you review fungi's jitsi meet config update change? I think we'll be able to test that tomorrow if we can have someone else sanity check it | 22:24 |
clarkb | In theory its a complete noop but a lot of moving pieces so good to have eyeballs | 22:24 |
ianw | will do | 22:24 |
fungi | yeah, it's really just updating our copied configs to match current stable upstream versions, and switching to their "stable" container image tag (which ironically is more recent than their "latest" tag) | 22:25 |
fungi | driven by the fact that turning on the pre-join page feature seems to stop firefox from thinking the audio stream is unsolicited, and that setting became an upstream default a year or more ago | 22:26 |
fungi | we've just been broken because we were frozen on increasingly ancient configuration | 22:27 |
clarkb | fungi: also maybe https://review.opendev.org/c/opendev/system-config/+/854678 tomorrow too? | 22:58 |
fungi | oh, yeah, for some reason i thought i'd already reviewed that one | 23:01 |
clarkb | I'm just looking at my backlog and making sure nothing is missed | 23:01 |
fungi | in that case, 852584 and topic:no-rebase could use a quick look from someone | 23:02 |
fungi | looks like you already reviewed most of those | 23:03 |
fungi | i meant to bring https://review.opendev.org/850061 up for broader discussion but i think i forgot to add it to the meeting agenda or raise it on the ml | 23:04 |
fungi | i guess that's why it's still wip | 23:04 |
*** dviroel|afk is now known as dviroel | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!