Wednesday, 2022-02-23

clarkbianw: I think it is using dhcp in ovh so I assume that is how it works in the gate00:00
clarkbI suspect that if we disabled dhcp in the gate we'd get better coverage since that only works if glean is sucessfukl00:00
clarkband ya I expected glean problems but then testing worked so I went with it :)00:00
clarkbthe MINA maintainer reports that MINA ssh client will use sha2 even if there is no server-sig-algs exchange. The RFC says this is correct. I'm really surprised openssh hasn't done that yet00:03
ianwso the NetworkManager bit of glean is working, but not reading from configdrive?00:07
clarkbianw: no I don't think any of it is beacuse rocky isn't recognized as a distro. It basically mounts and reads config drive then consults the distro name and exits00:07
clarkbianw: hwoeve,r when I run it by hand it exits 0 but noops. The systemd logs show it exited 100:08
clarkbI think we start with getting into shape adding rocky to the test list of distros. And take it from there00:08
clarkbI can look at this more closely tomorrow maybe00:08
ianwsigh that is really something i would have liked the gate tests to hit00:09
ianwa simple thing i guess is to check the service status in that test00:09
clarkb++ we can probably manipulate openstack in those jobs to disable dhcp and get better coverage of this00:09
clarkboh ya that would be a good check too00:10
ianw has rocky, perhaps we should just import that?00:11
ianwit is vendored00:11
opendevreviewIan Wienand proposed opendev/glean master: distro: sync to latest upstream version
Clark[m]Ya updating the distro lib is one piece then I'm glean/ we need to treat rocky as CentOS/rhel00:18
ianwwe can see if ^^ passes -- it looks like nothing major changed, but who knows00:19
*** dviroel is now known as dviroel|out00:20
ianwit does give a warning : glean/ DeprecationWarning: distro.linux_distribution() is deprecated. It should only be used as a compatibility shim with Python's platform.linux_distribution(). Please use, distro.version() and instead.00:21
ianwwhich makes things a bit weird, because distro was supposed to be a drop-in replacment for platform.linux_distribution() but now ... warns you to not do that00:31
opendevreviewIan Wienand proposed opendev/glean master: Add Rocky Linux support
opendevreviewIan Wienand proposed opendev/glean master: Remove
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Detect boot and EFI partitions in extract-image
opendevreviewIan Wienand proposed opendev/glean master: distro: sync to 3.6
opendevreviewIan Wienand proposed opendev/glean master: Add Rocky Linux support
opendevreviewIan Wienand proposed opendev/glean master: Remove
*** pojadhav|out is now known as pojadhav|ruck04:43
*** ysandeep|out is now known as ysandeep05:15
opendevreviewchandan kumar proposed openstack/diskimage-builder master: [DNM] RDO third party check job checking
*** bhagyashris_ is now known as bhagyashris06:28
*** amoralej|off is now known as amoralej07:06
*** jpena|off is now known as jpena08:34
*** ysandeep is now known as ysandeep|lunch08:49
*** ysandeep|lunch is now known as ysandeep09:24
*** bhagyashris_ is now known as bhagyashris09:25
gthiemongeHi Folks, there's a weird pip issue in one of the periodic jobs in Octavia:
gthiemongepip writes that pyroute2 "has inconsistent version: filename has '0.6.5', but metadata has 'file-.VERSION'"09:28
gthiemongeany ideas? corrupted package/mirror?09:29
fricklergthiemonge: this is broken for more than a week, the last working job installed pyroute2\<0.6, was there a requirements update?
gthiemongefrickler: yes, in the requirements repo:
fricklerthat matches the start of the failures. not sure why it would only fail for wallaby10:16
fricklercomparing to the xena job, I see two differences: pyroute2 seems to be capped to 0.6.4 in xena and it uses newer pip10:17
gthiemongefrickler: Ok, I can reproduce this issue locally with ubuntu bionic (the job builds a bionic image), but bionic is not a tested runtime for wallaby, I need to update the job to focal11:02
*** rlandy|out is now known as rlandy|ruck11:14
*** dviroel|out is now known as dviroel11:26
stephenfinLooks like Gerrit isn't sending emails at the moment11:28
stephenfinJust emailed openstack-discuss about it11:28
stephenfinLast email I received was Tue, 22 Feb 2022 11:57:11 +000011:29
stephenfin*was dated11:29
fricklerstephenfin: I received mail 30 mins ago, might be an issue with your provider, I can check logs in a bit11:48
fricklerfrom what I can see, it affects email. 550 Blocked by ivmSIP and/or ivmSIP/24 - see -
* frickler is not going to send an email to them as their delisting process requires, maybe someone from redhat wants to do that or some other infra-root11:52
fricklerinfra-root: slightly related, this is something we might want to clean up one way or other: 550 5.1.1 <>: Email address could not be found, or was misspelled (G8)11:59
stephenfinfrickler: Ack, I'll try find the relevant people to poke internally so. Cheers12:01
*** amoralej is now known as amoralej|lunch13:09
*** amoralej|lunch is now known as amoralej14:02
NeilHanlonappears the glean startup still failed for  (logs: - will take a look14:03
fungimgagne_: i bounced you a couple more messages from that e-mail thread just now, to keep you in the loop15:13
*** ysandeep is now known as ysandeep|dinner15:15
mgagne_fungi: thanks.15:31
*** gmann is now known as gmann_afk15:59
*** pojadhav|ruck is now known as pojadhav|dinner16:01
*** ysandeep|dinner is now known as ysandeep16:12
*** gmann_afk is now known as gmann16:17
clarkbfrickler: gitea seems to ahve confirmed that is a bug in their image diff with 1.16. It was added to the 1.16.3 milestone then removed from that milestone. Maybe that means they will try to fix it in 1.16.1?16:23
clarkbI think we can wait a bit and see if there is any movement on that before we upgrade. Though it isn't the end of the world as it appears to just be a rendering issue16:24
*** pojadhav|dinner is now known as pojadhav|ruck16:25
fungiyeah, and it's only going to come up in rare cases where someone looks at a diff which removed an image file16:25
clarkbrenamed or deleted image files yup16:25
fungiso i agree it doesn't seem like a blocker for upgrading, just something we'd like to see fixed as soon as possible16:25
clarkbI checked text files and they don't berak16:25
stephenfinfrickler: So I checked with IT, and they said admins need to be the ones to request removal from the blacklist16:43
stephenfin...which seems odd, given no one asked to be put on it, but that's what I was told16:43
fungistephenfin: neat. maybe one of the admins will find time to do it, but if red hat figures out how to remove things its employees want to receive from its own blocklist then that will likely be a faster resolution16:45
fungithe incentives seem to be woefully out of balance there16:46
stephenfinI know :( That's pretty much what I said16:46
fungithis is also why i highly recommend not using corporate-controlled e-mail accounts for public participation in open source communities16:48
fungithey're typically managed by admins who are primarily concerned with being able to send and receive messages with business partners, not public systems or individuals, so priorities are not aligned16:49
stephenfinYup, I have a personal email address I use for all other communities. This is likely as good a time as any to switch OpenStack'y things over to that now16:51
NeilHanlonfungi: wouldn't it be on the opendev side anyways? the blacklist says that `` is part of a spam-sending /24, which resolves to review02.opendev.org16:53
NeilHanlonor rather, vexxhost probably needs to request the removal since it's in their block16:54
clarkbI think the point is you should be able to solve it from both sides. I subscribe to a few bad actor lists on my firewall that filters dns records and does firewall filtering too. I can and have had to override subsets of those rules because I wanted to get the python documentation of all things.17:00
fungiNeilHanlon: i personally don't care, catering to every random blocklist on the planet only furthers the idea that relying on those flawed systems is a good idea17:00
fungiwe shouldn't need to seek permission from dozens of random companies on behalf of their customers17:00
clarkbPBS is another one that breaks due to their ad system17:01
fungiit takes the customers of those blocklists going back to them and complaining that they're blocking legitimate senders to actually reform anything17:01
fungiif red hat's employees find that red hat has made business decisions which render their company mail accounts less useful, then they should be pushing back on those choices made by their management. we didn't make that choice for them, and expecting us to solve it simply reassigns work from the people who should have responsibility for it17:03
fungiour responsibility is to keep our systems in good shape and make sure they only send legitimate messages to users, and i'm committed to doing that17:04
fungispam filtering systems which don't allow their users to indicate what senders are legitimate are inherently flawed non-solutions17:09
NeilHanlonfair enough17:10
NeilHanlonyeah, i've had to do similar myself, clarkb17:10
fungiyep, my personal spam filtering solution does take blocklists into account, but only uses them for weighted scoring of messages and not as a hard reject, so there has to be something pretty wrong with a message before i auto-discard it, even if its sender is also on several blocklists17:21
*** pojadhav|ruck is now known as pojadhav|out17:27
tbarron /set plugins.var.python.irssinotifer.api_token 1c124cc4-55c5-41fc-94e2-e15c7578705c17:28
*** jpena is now known as jpena|off17:35
fricklerthe other aspect of the whole spam story is that with our currently lack of bounce handling, I wouldn't be confident in claiming that isn't sending spam. not intentionally, but by what could be called lack of administrative oversight17:47
fricklerI'm wondering how other gerrit installations handle this17:48
*** amoralej is now known as amoralej|off18:07
*** ysandeep is now known as ysandeep|out18:20
fungiyeah, i'm not sure what we'd do with the bounces either, i expect there's a fairly high volume where people have switched e-mail addresses and not updated their gerrit accounts18:49
fungiwe could institute a policy of disabling accounts whose preferred addresses are bouncing back, though that seems a bit extreme18:49
NeilHanlonthere is so-called 'variable envelope return path' that's used by some stuff (mailman3, e.g.) to help detect bounces. not sure if gerrit supports it, though18:55
fungiyeah, we do verp in some places18:57
fungiwe can set up some solutions like that in the mta layer without gerrit needing to explicitly support it18:57
clarkbI think we already get the bounces18:57
clarkbthey are in the infra-root inbox18:58
clarkbwell not the inbox they get filtered elsewhere btui delivered there18:58
fungioh, right, we shuffle them into a subdir18:58
clarkbbut ya I'm not sure how we'd address that? just start killing accounts if they bounce email?18:58
NeilHanlonskip all the hard work and just delete accounts at random 😈18:59
fungican i just delete my account? then i can pretend the problem is solved! ;)19:00
NeilHanlonperfect! 19:00
clarkbthis is sort of why my long effort to clean up the notedb for conflicting accounts has fizzled out. YOu end up where there are no easy answers and then the easiest thing is to punt :)19:01
clarkbthe difficult thing here is if their email address bounces that is our contact method.. I suppose what we could do is disable any accounts for which that is the only email address listed. For those that have additional email addresses we can reach out to those email addrs and ask them to remove the offending email address19:03
clarkband after some time if we are ignored remove the offending email address ourselves. But that is a fair bit of work19:03
fungiyeah, so gerrit is currently using review@ for its from address and sender at the envelope layer (with reply-to of the individuals subscribed to the change), messages for which are sorted into INBOX.review19:05
fungii made the mistake of trying to open that...19:06
fungi-- Mutt: Directory [=INBOX.], File mask: !^\.[^.]                               19:06
fungiFetching message headers... 700/276063 (0%)19:06
fungisorry, accidentally grabbed a second line into my clipboard19:06
fungithink i'm a gonna just abort that19:07
fungii may have to put a bullet in ol' yeller here19:07
fungimutt seems to ignore keyboard interrupt in this situation19:08
clarkboh so those bounces are the ones that we're the reply to for?19:08
clarkbI guess that gives us an incomplete picture then and there are things to improve19:08
fungibounces normally go to the envelope senter19:08
fungireply-to is for actual replies, not bounces19:08
clarkbfungi: how are we getting them if we aren't the sender?19:09
fungiwe're the sender, just not the reply-to19:09
clarkbor at least I interpreted your initial message that review@ gets them all19:09
clarkband review@ != infra-root ?19:09
fungireview@ is aliased to that19:09
clarkbaha that explains it, thanks19:09
fungiand we have delivery rules putting messages for that address into INBOX.review19:10
fungiwhich is presently well on its way to having 300k messages in it19:10
clarkbgotcha so we do get a pretty good picture and could take action it? the problem is that the choices for action to take are all largely suboptimal?19:11
clarkboh you know what. These would almost have to be active accounts too19:11
fungiyes, and without clearing this and starting from inbox zero, it's going to be hard to guess how much of the bounce volume might be noise19:11
clarkbotherwise they wouldn't be generatign the events that send email in the first place. So these are active accounts with old email addresses19:11
clarkbI suspect a lot of it isn't noise given ^19:12
fungiwell, there are still reviews merging for people who actually died years ago, so it's not like there will be 019:12
clarkbya its nonzero but the bulk of it should be active accounts I suspect19:12
NeilHanlonlets just email everyone and tell them to make sure their emails are up to da---wait..19:13
fungiit's too bad we don't host this inbox on our servers. i could literally just move the messages aside or bulk delete them19:13
clarkbNeilHanlon: ya exactly :)19:14
fungii haven't killed mutt yet, it's loaded 4% of that mailbox so far19:21
fungithough i wonder if it will oom on me19:21
fungiyeah, i think it's going to, just watching `top` sorted by memory utilization. if i don't kill it, the kernel will19:22
clarkbianw: NeilHanlon: I've been digging into zuul reviews this morning but will try to page in glean stuff after lunch since I expect we'll be able to have more direct conversation about stuff. ianw in particular it seems the rework of how glean is executed fizzled out? We need to address that since a glean release will include it?19:26
opendevreviewClark Boylan proposed openstack/diskimage-builder master: Force use of NetworkManager with glean on Rocky Linux
opendevreviewClark Boylan proposed opendev/glean master: Add Rocky Linux support
opendevreviewClark Boylan proposed opendev/glean master: Remove
clarkbLets see if that is happier20:46
*** timburke_ is now known as timburke20:59
ianwclarkb: i think that we did all that on the glean-early stuff, but maybe just didn't release it21:07
clarkbianw: ya I think so too and it seems like the -src tests are testing it works? Do we need to make any changes on the dib simple init side to support it?21:08
ianwno i don't think so21:10
ianwstephenfin: so ultimately you got no help from redhat IT?  can you send me a link to any issue etc.21:11
ianwmy overnight inbox is weirdly empty21:11
clarkbianw: the big indicator to me that we are probably fine is that the rocky test failed against those glean changes. This indicates it is doing stuff to test it and if the other distros pass then we are probably ok21:12
ianw++ the NetworkManager bits you've added definitely seem on the right track21:13
*** dviroel is now known as dviroel|out21:18
ianwinfra-root: while i agree with all the points about it being stupid, i have also sent a mail to try and get the server de-listed21:24
clarkbianw: thanks21:24
fungithanks ianw!21:28
NeilHanlonthank you ianw ! 21:30
clarkblooks like the glean rocky change passes the rocky test now with NetworkManager in use22:07
clarkbI don't know that the ordering matters too much here since rocky isn't gating anything yet. But the depends-on is probably fine as is22:08
clarkbanyway school run, but I think we can work to land some of that stuff next22:12
ianwclarkb: 3.6 is a typo, it's v1.6.0 as you say.  i don't know if we want to retest for that22:15
priteauHello. A kayobe patch earlier went through gate jobs successfully but didn't merge, yet I cannot see anything blocking it:
Clark[m]I'm not worried about it or would've -1'd22:20
ianwpriteau: hrm, interesting, i agree nothing obviously stopping it merging22:24
priteauianw: some event lost in zuul?22:29
ianwit's parent merged a long time ago
fungipriteau: ianw: a later revision of its parent merged. gerrit won't allow 827486 to merge without a rebase22:33
fungithe orange-ish tint to the "(Merged)" in the relation chain list is the color-blind-useless clue the gerrit webui gives you for that22:34
ianwoh, doh, yep22:35
priteauI see, PS2 is the parent22:35
priteauthanks for the tip22:35
fungihopefully when we get gerrit's mergeability checks turned back on, we'll also see the "can't merge" message listed in the ui for such changes22:36
ianwspeaking of gerrit, i seem to be getting mail again22:37
fungioh good22:38
sshnaidmfungi, hi, do you know why it may happen - I'm core in the repo, but I can't merge and +2 into stable branch in this repo, which I created before:,branches23:07
sshnaidmand I can't merge this patch
clarkbsshnaidm:,members this group has permissions on the stable branches for that repo and appears empty23:09
sshnaidmclarkb, interesting.. and how can I add there? (can I?)23:10
clarkbuh I think the stable team has historically managed that? I don't know if that is important for this repo though23:10
clarkbI guess in this case it isn't related to the openstack stable releases so maybe doesn't matter?23:11
sshnaidmclarkb, can I just replace ansible-collections-openstack-stable-maint by ansible-collections-openstack-core23:11
clarkbsshnaidm: yes that is one way to handle it23:11
sshnaidmcool, will add a patch..23:11
clarkbMostly I just awnt to make sure whatever we do doesn't get in the way of the stable team which has historically managed this stuff for the openstack stable stuff23:11
clarkbsshnaidm: I think you can just delete the block I highlighted to do what you describe23:12
sshnaidmthe whole "refs/heads/stable/*" ?23:12
sshnaidmso that the main block apply for all?23:13
clarkbyes, but again we should double check with the stable team before doing something like that. I suspect it is fine since they care about the integrated stable release and this is separate23:13
sshnaidmit's separate and it's SIG repo23:14
sshnaidmand it didn't have any stable branch till prev week23:14
clarkbya so should be fine to remove that block then23:14
clarkband just rely on the refs/heads/* access block instead23:15
opendevreviewShnaidman Sagi (Sergey) proposed openstack/project-config master: Remove special config for stable branch in collections repo
sshnaidmclarkb, fungi please review ^23:17
*** rlandy|ruck is now known as rlandy|ruck|bbl23:18
clarkbok the glean stack has my +2's maybe fungi can take a look?23:25
fungiwhat's the review topic?23:25
opendevreviewMerged openstack/diskimage-builder master: Force use of NetworkManager with glean on Rocky Linux
opendevreviewMerged openstack/project-config master: Remove special config for stable branch in collections repo

Generated by 2.17.3 by Marius Gedminas - find it at!