Tuesday, 2022-01-18

ianwlooks like the -devel job failure is actually picking up an ansible issue - https://github.com/ansible/ansible/issues/76779.  i need to debug a bit more00:17
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Call grub2-switch-to-blscfg with explicit config  https://review.opendev.org/c/openstack/diskimage-builder/+/82465900:34
*** odyssey4me is now known as Guest82200:38
fungiooh, i love it when our find bugs in someone else's software00:56
fungier, when our jobs find bugs00:56
*** tobias-urdin5 is now known as tobias-urdin02:09
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Call grub2-switch-to-blscfg with explicit config  https://review.opendev.org/c/openstack/diskimage-builder/+/82465902:56
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Don't install python3-virtualenv on centos9  https://review.opendev.org/c/openstack/diskimage-builder/+/82503403:24
*** artom__ is now known as artom03:28
*** frenzy_friday is now known as frenzyfriday|ruck04:36
*** ysandeep|pto is now known as ysandeep05:23
*** anbanerj is now known as frenzyfriday|ruck07:00
zbrwdyt about the direction in which https://github.com/pypa/pip/issues/10556 is going? My impression is that pip folks seems to be too sure that they know everything about container building.07:39
*** ysandeep is now known as ysandeep|lunch08:03
*** jpena|off is now known as jpena08:35
*** ysandeep|lunch is now known as ysandeep09:11
opendevreviewSandeep Yadav proposed openstack/diskimage-builder master: Add rhel9 in grub2 element  https://review.opendev.org/c/openstack/diskimage-builder/+/82507009:16
*** odyssey4me is now known as Guest92509:22
*** ysandeep is now known as ysandeep|brb11:05
*** rlandy|out is now known as rlandy|ruck11:14
*** ysandeep|brb is now known as ysandeep11:14
*** dviroel|afk is now known as dviroel11:25
opendevreviewAurelien Lourot proposed openstack/project-config master: Mirror charm-nova-compute-nvidia-vgpu to GitHub  https://review.opendev.org/c/openstack/project-config/+/82508911:27
fungizbr: i've been closely following pep 668 and they do seem to want to find solutions for containers as "external package managers" it's just that there are a lot of competing interests to consider13:26
fungifor our purposes, we've been going out of our way to at least run everything as non-root users within our containers, though i expect we do still pip install some things in them as root13:27
fungi(though maybe not, we may be using a non-root user when executing container image builds too, i'd have to look)13:28
opendevreviewSzymon Datko proposed zuul/zuul-jobs master: Add support for RPM packages  https://review.opendev.org/c/zuul/zuul-jobs/+/82511815:00
zbrfungi: sometimes using non-root user can probe a real PITA. It seems that in the end they are willing to reconsider it, even if in a bit weird way with an imposed delay of 3 months.15:23
zbrsomehow i have the impression that pip will give the root warning even in chroot or if with `--no-deps` which are two other reasons why that warning is overzealous.15:25
fungiyep, but at least it _is_ merely a warning/reminder15:34
opendevreviewTristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to include ChangeReady event  https://review.opendev.org/c/opendev/system-config/+/82513115:49
*** ysandeep is now known as ysandeep|out16:04
clarkbI'm going to approve https://review.opendev.org/c/opendev/system-config/+/823026 now and we can sort out a restart of gerrit later today after that has merged16:18
clarkbThis is the 3.3.9 bump16:19
*** dviroel is now known as dviroel|lunch16:21
clarkbre pip I think there are bigger issues when it comes to giving people unhelpful info in pip output. Particularly the combo of unreliable CDN and constraints saying it can't find a valid version could be far more clear. A warning explaining that pip as root is not recommended seems reaosnable since it isn't recommended and does have problems when you do that16:33
clarkbfungi: fwiw I think many of our container image builds install as root before switching the user16:33
opendevreviewGustavo Sanchez proposed openstack/project-config master: Mirror cinder-solidfire and cinder-nimblestorage to Github  https://review.opendev.org/c/openstack/project-config/+/82513816:37
opendevreviewClark Boylan proposed opendev/system-config master: Install Limnoria from upstream  https://review.opendev.org/c/opendev/system-config/+/82133116:40
clarkbfungi: ianw  ^ I looked closer at limnoria and while they don't seem to have recent proper releases they do checkpoint tags of master and there is one fromthe first of the year. I think we start there rather than master proper16:41
fungiyeah, part of the unhelpful messaging from pip is that it silently ignores sdists with build failures and tries older and older versions until it gets one to build, which can lead to you running much older versions of dependencies than expected if you don't have a sufficiently high lower-bound16:49
artomHow would I check whether Zuul can use a particular ansible module? ansible.posix.mount in this case specifically, but the question can remain general17:03
clarkbartom: it can use anything you get out of a `pip install ansible` then there are a couple of additional things like the openstack and kubernetes modules too iirc. But generally anything you get out of pip installing ansible should work17:04
fungijust not on the executor17:04
artomclarkb, huh cool, I remember reading somewhere that Zuul's ansible was limited17:04
artomfungi, ah, that was it, it's limited on the executor17:05
clarkbright well it is limited as fungi points out but the modules are there. Depends more on where you execute them17:05
fungiartom: i assume your question is about *our* zuul deployment, not about your own zuul install somewhere?17:05
artomfungi, heh, yes17:05
artomYeah, this is being executed (well, attempted to be executed, because it's complaining of a module not found) on the target CI VM.17:06
clarkbI suspect the issue is more that you are trying to use modern ansible collection syntax with zuul and it hasn't updated its ansible for a while (focus has been on scale out scheduelr stuff but expect ansible updates after 5.0 happens)17:06
fungiartom: more generally, you can find what zuul shadows by default from its ansible installation on the executor here: https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible17:06
clarkbhttps://docs.ansible.com/ansible/2.9/modules/mount_module.html is probably better doc for that module17:07
clarkb(note it doesn't use the collection namespacing stuff)17:07
* artom goes to figure out what this newfangled ansible collection stuff is, and why the same module can be references as two different things, and which one will work is apparently a crapshoot?17:09
clarkbartom: it was/is a whole thing. Ansiblefest 2019 red hat/ansible said "we're doing this" and then everyone has been trying to get up to speed since. But basically they took ansible and split it up into many many repos as collections of ansible modules. A number of them are included by default when you install ansible but many others are not and have to be installed separately17:10
clarkbartom: I don't think absible 2.9 understands the syntax of ansible.posix.mount. It is just 'mount' on that version of ansible17:11
*** marios is now known as marios|out17:11
* artom apologises on behalf of his employer.17:11
clarkbbut the collections have that namespacing because you might install a collection that is artom.tools.mount ot whatever17:11
jrosserin later ansibles there is a mapping between the previous unqualified name and the fully namespaced equivalent17:11
jrosserso it should work either way17:12
clarkbjrosser: right but only on newer ansible which zuul doesn't support yet17:12
fungiahh, so continuing to use "mount" should be forward-compatible, at least for a while17:12
clarkbat least that was my understanding17:12
clarkbfungi: ya17:12
jrosserfungi: yes old code will still work17:12
fungijust not the other way around is, i guess, what clarkb was getting at17:13
fungiso "it should work either way" _on newer ansible_ but if you use the old syntax you'll be able to make it work on old and new ansible17:14
jrosseryes, i wasnt too specific there :/17:14
*** dviroel|lunch is now known as dviroel17:19
*** jpena is now known as jpena|off17:33
opendevreviewMerged opendev/system-config master: Update Gerrit images to 3.3.9 and 3.4.3  https://review.opendev.org/c/opendev/system-config/+/82302617:39
opendevreviewMerged opendev/system-config master: Update gerritbot-matrix version to include ChangeReady event  https://review.opendev.org/c/opendev/system-config/+/82513117:46
ianwwhat is the chance that zuul autoholds aren't autoholding as much as we'd like?19:35
ianw0000000000 | openstack | opendev/system-config | system-config-run-review-3.4 |     .*     |     1     | ianw: gerrit-3.4 hold 19:35
ianwi put that in yesterday19:35
clarkbianw: did the change not fail maybe?19:35
ianwhttps://zuul.opendev.org/t/openstack/build/412485bff9ff4b44b1f67436ad0f9f67/ was the job i'd expect to hit it19:36
ianwyeah i got distracted on ansible issues yesterday, i'll run it through now and if it doesn't hit can debug in a more immediate fashion19:51
opendevreviewIan Wienand proposed opendev/system-config master: [dnm] force fail of gerrit-3.4 jobs  https://review.opendev.org/c/opendev/system-config/+/81593319:52
fungimy workstation crashed toward the end of the meeting, but i'll start bringing up and updating the jitsi servers once i've got it back online19:57
fungiokay, got sidetracked with some deferred package updates since i wanted to make sure i took the opportunity to update the kernel as well, but working on meetpad bits now. i'll start with the jvb servers20:50
ianwok, it appears the gerrit job hasn't held itself again21:01
clarkbexciting. corvus ^ fyi I half wonder if that is an sos bug in zuul21:01
fungianything useful about it in the debug log?21:02
clarkbhttps://etherpad.opendev.org/p/9Cy4GFmrFCK1yLpxgs5V does a gerrit downtime/maintenance/upgrade notification need more info than that?21:19
fungilgtm. i added a link to the renames list21:23
*** dviroel is now known as dviroel|out21:26
corvusi think the autohold thing is a slight behavior change in zuul-client compared to the old zuul command.  requires specifying the full canonical repo name now.21:29
clarkbin this case opendev.org/opendev/system-config ?21:30
fungii've upgraded all the jitsi container images on our meetpad/jvb servers and started services back up, if anyone wants to test it out (i'm not currently able to do a/v testing myself, but poked at the browser-accessible features at least and everything seems to be functional)21:31
fungiif anyone else wants to test, please do so21:31
fungii'll hold off on the status log for it until folks have a chance to confirm it seems to be working21:32
clarkbif anyone else has a mic working I can hop on a call21:32
corvussure, let me fix my hair21:33
fungii can see your video stream21:35
fungiand it didn't kill the shared document, that's a good sign21:36
fungionce someone with working mic/speakers joins you'll hopefully be able to confirm audio is working21:36
clarkboh sorry grabbed a glass of water. I'll join momentarily21:38
corvusomg reaction sounds21:38
fungii can tell my browser was trying to play some sound effect, just not the audio stream. i need to dig into what the deal is with my system21:39
fungiit has "polls" now too, is that new?21:39
fungiand "break out rooms"21:39
fungii probably just wasn't paying close attention to available features in the past21:40
clarkbfungi: note it says I am joining muted but then when I join I am not muted21:41
fungioh, neat21:41
fungii wonder if something changed with how we need to configure that21:41
fungiwe set it to mute video on joining, is that what you're talking about?21:41
clarkbyes that21:44
clarkbit does give me a pop up saying I'm starting muted but then I am not21:44
clarkbwhich is probably more problematic21:44
fungistarting with your camera muted, specifically?21:46
fungi(not audio?)21:46
clarkbno audio. I don't have a camera21:46
fungiahh, i thought we configured it to only mute video on joining, not audio21:46
clarkbfungi: yup the .env file on meetpad agrees. I think the regression is that I get a pop up saying I'm starting muted21:48
clarkbBut maybe it means my video is muted and I've misread it. I'll reconnect and read carefully21:48
fungiwhereby it means muted video but doesn't specify maybe21:48
fungiokay, but you were also able to see corvus's video, and the two of you could hear one another?21:49
clarkb"You have started the conversation muted" is what it says so it is ambiguous21:49
clarkbyes I could see corvus' camera and we could hear each other21:49
fungii only get the join/part/reaction sound effects, but am fairly certain it's my browser or pulseaudio21:49
clarkbI think we should consider setting the setting to start audio as muted too to make that ambiguity of the message less painful. But I don't think this is regression21:49
fungi#status log The meetpad.opendev.org services have returned to working order, and are available for use once again21:51
opendevstatusfungi: finished logging21:51
opendevreviewClark Boylan proposed opendev/system-config master: Start meetpad meetings with muted audio  https://review.opendev.org/c/opendev/system-config/+/82517921:55
clarkbI think ^ will make the muting pop up far less ambiguous and confusing21:55
clarkbI'll send out the gerrit maintenance announcement momentarily21:57
clarkbcorvus: for the restart to pick up https://review.opendev.org/c/zuul/zuul/+/824864/ there isn't any reason to coordinate that with our gerrit update to the latest 3.3.9 image right? We can do that one as a rolling restart?22:17
corvusclarkb: correct; i'm also thinking maybe we should get 825042 and 825079 in.  maybe 824706 as well.22:24
clarkbcool I'll proceed with the gerrit restart when I'm ready for that. Probably in 45 minutes or so. Warning the openstack release team now22:25
clarkbI'll also look at those changes22:25
clarkbinfra-root there are a few tripleo changes that look close to merging. I'll give those a few minutes to clear out then docker-compose pull && docker-compose down && docker-compose up -d on review0223:02
*** lajoskatona_ is now known as lajoskatona23:04
clarkbdown to one job now holding up a bunch of potential merges :)23:10
clarkbif I had known the zuul estimate wasn't too accurate I could've been done by now :) but better safe than sorry when it takes that long to gate those changes23:18
clarkbthe job is in the post run activities now looks like23:28
clarkbthese tripleo jobs spend a lot of time just moving data around at the end of a job :/23:32
fungidue to the sheer number of "log" files saved23:32
clarkb2022-01-18 23:07:43.952093 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] <- I think that means I don't need to care about it23:33
clarkbI went ahead and ran a pull. mariadb and gerrit images updated. The gerrit image matches what I see in docker hub from today's update. Any objection to a down and up -d now?23:34
clarkbsounds like no? I'll proceed with the gerrit update to 3.3.9 then23:36
clarkbactually let me start a screen23:36
clarkbok screen started on review. Proceeding with restart in that screen23:37
clarkbweb ui is up for me and I see the expected 3.3.9 version in the footer23:39
clarkb#status Log upgraded Gerrit on review02 to 3.3.923:43
opendevstatusclarkb: finished logging23:43
clarkbI don't have any changes to push right now but it seems to be working on the web side23:43
clarkbianw just pushed a change to zuul and that was detected by matrix gerritbot23:44
clarkband now zuul has merged a zuul change through gerrit. All looking good there23:47
*** rlandy|ruck is now known as rlandy|out23:50
clarkbI have closed the screen23:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!