Wednesday, 2022-11-09

opendevreviewMichael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts
opendevreviewMerged zuul/zuul-jobs master: Make test-registry-buildset-registery-k8s-<docker|crio> non-voting
opendevreviewMerged zuul/zuul-jobs master: ensure-podman/skopeo : update for Jammy
opendevreviewMerged zuul/zuul-jobs master: ensure-pip: use dummy package for testing
opendevreviewMerged zuul/zuul-jobs master: linter: Use capitals for names
opendevreviewMerged zuul/zuul-jobs master: Fix ansible-lint name[template]
opendevreviewMerged zuul/zuul-jobs master: Add names to include tasks
opendevreviewMerged zuul/zuul-jobs master: Standarise block/when ordering
opendevreviewMerged zuul/zuul-jobs master: Update to ansible-lint 6.8.2
opendevreviewMerged zuul/zuul-jobs master: zuul-tests: autogenerate some more roles
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Combine ensure-pip playbooks into a test role
ianwclarkb: just a bit confused trying to follow, but also probably haven't fully recovered everything that goes on there from my mental swap partition :)04:09
ianwcomment inline04:09
Clark[m]ianw: the first step we do with the build tool makes a single wheel for the package we want to install. My understanding of the `pip wheel` command is that it is supposed to make wheels for everything necessary to install the item(s) listed04:16
Clark[m]So what that second step does is say make wheels for all of our deps04:17
Clark[m]And according to the pypa/pip upstream this is the correct way to have pip make wheels for everything you need to install instead of relying on the cache04:17
Clark[m]So you can give `pip wheel` a package name, an sdist, a requirements file, a wheel file etc and it is supposed to sort out wheels for everything from that04:19
*** yadnesh|away is now known as yadnesh05:28
ianwthanks for clarifying.  i indeed had a play with it and agree, all lgtm05:37
*** marios is now known as marios|ruck05:57
timburkeanybody else seen something like this lately when running `git review --setup`? seems i can't copy the commit hook
timburke(i mean, *i* can always copy it from one of the many other local checkouts i've got that's already set up -- but i pity the poor newcomers...)06:12
fricklertimburke: can you run the scp command manually? for me this is still working fine, fwiw06:36
timburkefrickler, nope; same (slightly ominous) "subsystem request failed on channel 0"06:37
timburketacking on a -v and only looking at the bits following "Server accepts key":
fricklertimburke: what OS and ssh version is this? the XMODIFIERS env looks weird and I also only see "pledge: filesystem" in my run, lacking to "full"06:46
timburkefrickler, Fedora 36, OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 202206:48
timburkenot sure how i picked up XMODIFIERS, but unsetting it things fail the same way06:50
fricklerwas there some recent change to those versions, which might be related? On Jammy I have "OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022"06:51
fricklerother than that maybe some issue with your network environment. do you have a chance to test from a different host with a different OS?06:54
timburketrying on a jammy VM, things are fine -- must be something with those version differences :-/06:57
timburkerelevant openssl changelog:
timburkeand openssh:
timburkeincreasing the verbosity (and dropping XMODIFIERS):
fricklertimburke: ah, that ssh reno mentions switching to sftp. if I do that by adding "-s" to the scp command, I get the same error. so maybe fedora switched the default already. can you check whether you have a flag to switch back to scp mode?07:05
timburke-O seems to have done it --
timburkethanks frickler~07:10
noonedeadpunkhey there! Would be great to get landed this week as we want to make stuff in time for Zed (we're trailing so need to release until 16 of December). 07:13
*** yadnesh is now known as yadnesh|afk08:09
jrossertimburke: does this help in newer git-review?
fricklerjrosser: at least that seems to match the issue exactly, thx for the pointer08:49
*** yadnesh|afk is now known as yadnesh08:53
*** dviroel_ is now known as dviroel09:40
opendevreviewTama McGlinn proposed opendev/git-review master: Add message option
opendevreviewTama McGlinn proposed opendev/git-review master: Add message option
opendevreviewTama McGlinn proposed opendev/git-review master: Add message option
opendevreviewTama McGlinn proposed opendev/git-review master: Add message option
opendevreviewDr. Jens Harbott proposed opendev/git-review master: Fix nodesets for tox and add py310 job
fungifrickler: timburke: yes, openssh deprecated the old scp protocol and expect "modern" scp servers to speak sftp protocol now12:43
fungiand yeah, that should be addressed with git-review>=2.312:44
fricklerfungi: the py310 tests for git-review that I tried to add above are failing, likely some request mocking needs to be fixed, do you want to have a look at that?12:59
frickleralso interesting coincidence to see so much git-review action happening on the same day13:00
fungifrickler: i had started an attempt at updating to newer gerrit/java/ubuntu for git-review testing, but haven't made a lot of progress. if it's a matter of older gerrit not running on new enough javascript then that may need to be picked back up13:03
fungithere is no "mocking" in the git-review tests. they actually install and run a gerrit server13:03
fungibut since that test setup has to bootstrap a working gerrit account to test with, and that breaks in various ways in newer gerrit versions13:04's nontrivial13:05
fricklerfungi: ah, o.k., then I'll drop py310 from my patch again for now and just fix the CI13:05
fungi was as far as i got with it13:06
opendevreviewDr. Jens Harbott proposed opendev/git-review master: Fix nodesets for tox jobs
fungii guess i could push it forward to the version of gerrit we're running now that the ssh host key issue is addressed13:07
fricklerI was just thinking whether the failures might be due to new openssh in 22.04, then, that sounds plausible13:08
fungioh, that will almost certainly break for the gerrit version we test with there unless some workarounds are added, yes13:10
fungior unless we can get the account setup worked out for a recent gerrit version13:10
fricklerhmm, why does gerrit offer commits from different projects to choose from in the rebase menu? doesn't make any sense at all to make. trying to rebase on top of 86410613:16
opendevreviewDr. Jens Harbott proposed opendev/git-review master: Add message option
fungii've not noticed that, but i also don't recall ever trying that rebase feature13:40
*** slaweq_ is now known as slaweq14:09
*** dasm|off is now known as dasm14:10
opendevreviewVishal Manchanda proposed zuul/zuul-jobs master: [DNM] Test horizon with removing install chromium task
*** dviroel is now known as dviroel|lunch15:50
clarkbfrickler: I confirm that rebase behavior when you switch to rebasing on a specific change. That seems worth filing a bug for. I can do that if you prefer (I already have an account set up and everything)16:01
clarkb added beta local recording support to jitsi meet16:17
clarkbgmann tested this and reported there is an echo in the audio.16:17
clarkbI think recordrtc as a browser plugin is another option that people can try (this is what I thought gmann had tested initially as i didn't realize jitsi meet had recently added this featre)16:17
fungineat, and good to know they added client-side recording built in, even if it's still suboptimal for the moment16:22
jrosserclarkb: i just tested rescue here and it works for me on bvf instance using two completely different images, centos and ubuntu16:26
clarkbjrosser: do you have to set any properties on the image you use for rescuing? melwitt pointed me to the tempest testing for rescue and in the bfv case it sets a bus and device type on the image16:27
jrosserand i have "hw_rescue_bus='scsi', hw_rescue_device='disk'" properties on all images16:27
clarkbyup I suspect ^ is what is needed16:27
clarkbjrosser: thank you for testing that. I think that points us in a direction that should hopefully get this working cc mnaser__ 16:29
jrosserthis is how it looked inside the ubuntu that rescued the centos
*** marios|ruck is now known as marios|out16:30
clarkbjrosser: the other thing to check is `blkid` to see if you have colliding labels and your grub config for any root label specifiers. I suspect that this isn't an issue though as those filesystems seem to have done the correct thing16:30
opendevreviewVishal Manchanda proposed zuul/zuul-jobs master: Remove "Install chromium-brower" task to fix CI.
clarkbjrosser: thanks that confirms no colliding label values16:32
clarkbthat ubuntu image must be booting by uuid instead16:32
clarkbanyone know if Visahl is on IRC or matric re 864119?16:33
fungiclarkb: vishalmanchanda in #openstack-infra and #openstack-horizon16:34
fungii think we discussed that problem last week, or at least i remember looking into the job failures16:35
fungiyeah, it was 2022-11-02 12:24:15 in #openstack-infra16:36
*** yadnesh is now known as yadnesh|away16:42
*** dviroel|lunch is now known as dviroel16:46
fricklerclarkb: yes, please create a bug report, thx16:49
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup uwsgi build hacks
opendevreviewClark Boylan proposed opendev/system-config master: Switch python-builder/python-base to pip wheel
clarkbianw pointed out that a recent uwsgi release claims to ahve fixed the issues I was trying to workaround with the old change17:10
clarkbI've removed the workarounds and the pip wheel change should exercise uwsgi builds17:11
opendevreviewClark Boylan proposed zuul/zuul-jobs master: Install dbus-user-session along with xvfb and firefox
*** dviroel is now known as dviroel|biba19:51
*** dviroel|biba is now known as dviroel|biab19:51
opendevreviewJeremy Stanley proposed opendev/system-config master: Add summit-track-chairs ML in place of old name
fungiinfra-root: can i get expedited, the conference organizers are wanting to start using the new ml today so i'll take care of the manual steps after that's deployed20:16
ianwdoes removing the list delete it?  i see you plan to manually migrate subscribers20:19
fungime running `rmlist` deletes it, but i don't want to do that until the change deploys so that ansible won't recreate it20:20
fungii've had that happen to me more than once, learned my lesson eventually20:20
opendevreviewMerged opendev/system-config master: zuul reboot: pin to
opendevreviewMerged opendev/system-config master: Remove
clarkbya the ansible stuff for mailman doesn't know how to delete things just create them20:35
clarkbfungi: is it ok for me to approve that change and have ansible create the new list?20:36
fungiclarkb: yes, that's the plan. let ansible create the new list and then i'll diff the list configs and set similar options on the one one and move stuff over20:37
fungifeel free to approve if you're good with it20:37
clarkbI wasn' sure if you wanted to create it first too and have ansible simply reflect that config. Approved now20:38
fungii considered doing it that way, but the more i thought about it this was simpler20:38
opendevreviewIan Wienand proposed opendev/system-config master: inventory: add host keys
opendevreviewIan Wienand proposed opendev/system-config master: bastion host: add global known_hosts values
ianw^ does anyone have thoughts on that?  I feel like we've discussed it before ... but perhaps not in the context of the bridge bootstrap steps?20:58
fungii've suggested it in the past, and am wholly in favor of the idea20:59
fungithe alternative was relying on sshfp records, but you need custom client configuration to have openssh auto-accept unknown host keys based solely on sshfp21:00
fungii suppose we could add it in system-config the same time we add the ip addresses to the inventory21:01
ianwyeah and istr we hit issues with ports and sshfp records21:01
ianwthat approach is perhaps different to others because it reads the inventory in the bridge bootstrap and writes the values from that; rather than actually being in the nested ansible when *ansible* has read the inventory21:02
fungitrying to have more than one sshfp record for different ports yes, and another problem is that openssh won't auto-update known hosts if the host key associated with a name changes (e.g. after server replacement) so sshfp is complicated in that case too21:02
ianw... that's a really interesting idea about essentially a dynamic known_hosts ... i wonder if you can do that21:04
opendevreviewMerged opendev/system-config master: Add summit-track-chairs ML in place of old name
fungiianw: one thing which rubs me the wrong way there is the embedded hostnames in the entries. we should probably be populating those from the inventory?21:06
fungii know it complicates the role though21:07
funginot a blocker, but it seems... redundant21:07
ianwquite possible -- really that's an artifact of me running ssh-keyscan and > the output to a file, then running an emacs macro that grabbed 3 lines from that and pasted them in below the hostname21:08
ianwso basically that was the raw output from ssh-keyscan21:08
fungiand if they don't match 100% then it will be useless21:08
fungioh, also renames will still be a problem unless we configure the client to not write its own known_hosts entries as well21:10
fungibecause by default, it's going to go "okay i recognize this host key matches the name i'm connecting to, and for extra safety i'll also write my own copy of it with the ip address so that it's recognizable if i don't connect by name some time later"21:11
ianwyeah i was hoping to basically remove /root/.ssh/known_hosts21:11
fungiwe can mitigate that by also embedding the ip addresses in the host records, but at that point you're definitely going to want to dynamically generate them from other inventory data21:11
fungiotherwise, you'll end up with stale (redundant) entries in /root/.ssh/known_hosts which won't get updated when we change or replace entries in /etc/ssh/ssh_known_hosts and then the client will quite likely refuse to connect21:13
fungithough that's also worth testing, i suppose21:13
ianw++ i think that's a good point to ditch those hostnames in there, and write out dynamically the hostname, and potentially public_ipv4,public_ipv621:14
fungii'm honestly not sure what happens if ssh finds host keys in both files and they're completely different keys21:14
ianwi was thinking the /etc because sometimes admin users will bounce through, and so it should allow connecting for everyone21:15
fungiyeah, i like it for that reason as well21:16
ianwthe launch-node on jammy can give us the right values now too.  the bionic era ssh-keyscan didn't have the flags to output the values21:20
corvusthis all sounds great for in-system -- i like sshfp for remote use (ie, our initial connections to servers from our workstations)21:41
corvus(so i think we should keep putting those records in dns for that reason)21:42
opendevreviewJames E. Blair proposed opendev/system-config master: Add ssl support to zookeeper-statsd and fix latency handling
corvusi think that will solve the longstanding mystery of our zero-latency zk cluster :)21:46
clarkbcorvus: are some stats only available over ssl?21:48
clarkbianw: I think the host key stuff is fine, the only thought I've got is we should update our launch node tooling and/or readme to make it easy to update that file too.21:50
clarkbwe already write out the inventory file contents, maybe update that to emit the host key data in the block as well?21:50
corvusclarkb: no, that's a 2-in-1 change: 1) add ssl; 2) fix type handling.21:51
clarkboh i see21:51
corvusthey are independent, but when i tested that locally it was as a unit.  seemed harmless to leave them combined.21:52
fungicorvus: agreed on also keeping the sshfp records, my point was they don't solve the "our automation needs to validate the keys" problem, but as long as we don't mind adding host keys to the inventory and dns, having both makes things smoother21:55
fungiwe have to add stuff to both files anyway when setting up a new server, so it's already "copy/paste this blob"21:56
fungi(both files being the ansible inventory and the bind zonefile)21:56
fungioof, that bridge change which merged has triggered 32 deploy jobs21:58
fungithis will be a while21:59
ianwfungi: and I think are getting pretty close to being able to run 31 of those in parallel22:11
ianwmaybe not quite; there's still other dependencies with letsencrypt etc.  but more parallel than one at a time, anyway :)22:11
ianwcorvus: lgtm but is there a reason to not just make it a float always?22:19
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] trigger openafs testing
corvusianw: good q.  all of the metrics are integers except one, and i honestly don't recall if the statsd-graphite part of the system would behave differently or not there.  seems like there's a possibility we could end up with a 1 ~= 0.9999999 situation.  in other words, i'm being conservative.22:45
ianwok, seems fine 22:48
ianw... in things that are not fine, openafs- is indeed failing to build on 9-stream; which is holding up the wheel release22:48
ianwi bet this is fixed in 1.8.9pre1 ... but pre releases have not been great for us in the past22:49
ianw...;a=commit;h=7a181415db1b2142d125714f1dea32879e2ca07d 22:51
opendevreviewNikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms
*** dasm is now known as dasm|offp23:03
*** dasm|offp is now known as dasm|off23:03
clarkbianw: did centos 9 get a new kernel?23:09
clarkbseems odd that it owuld break otherwise23:09
clarkbalso left a question inline23:11
opendevreviewIan Wienand proposed opendev/system-config master: openafs: copy dkms log directory
clarkbianw: on the etherpad syslogging change what ensures the docker-foo tagged log lines end up in /var/log/containers? Is that just config we put on all hosts?23:13
ianwclarkb: yep, ensure-docker does that ->
clarkbI have confirmed the logs are there in the test job so change lgtm23:14
opendevreviewNikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms
opendevreviewNikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms
opendevreviewNikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms
opendevreviewIan Wienand proposed zuul/zuul-jobs master: enable-kubernetes: Fix jammy install, improve pod test
opendevreviewMerged zuul/zuul-jobs master: test-registry-post: collect k8s logs
opendevreviewMerged zuul/zuul-jobs master: ensure-kubernetes: move testing into common path
opendevreviewMichael Kelly proposed zuul/zuul-jobs master: helm: Add job for linting helm charts

Generated by 2.17.3 by Marius Gedminas - find it at!