Monday, 2021-12-13

opendevreviewIan Wienand proposed opendev/system-config master: infra-prod: fix infra-prod-service-zookeeper soft dependency
fungiianw: thanks for running that down!00:04
opendevreviewMerged opendev/system-config master: infra-prod: fix infra-prod-service-zookeeper soft dependency
ianwheh well i created the problem :)00:18
fungiyou can't make config omelette without breaking config eggs00:38
opendevreviewIan Wienand proposed opendev/system-config master: zuul-*: use multiline formatter
*** ysandeep|out is now known as ysandeep04:20
*** frenzy_friday is now known as anbanerj|ruck04:35
ianwok, LE periodic job has run again :
*** bhagyashris_ is now known as bhagyashris06:21
*** ysandeep is now known as ysandeep|intv06:25
*** ysandeep|intv is now known as ysandeep08:51
*** ysandeep is now known as ysandeep|lunch08:58
hrwis there some doc about migrating to other SSO login?09:15
hrwI managed to lock myself out of so will need to create new account there and regain kolla permissions09:16
*** ysandeep|lunch is now known as ysandeep09:21
fricklerhrw: afaict if you lost access to the old login, things get difficult. but other kolla folks should be able to give access to the new account once you created it09:37
hrwok, thanks09:37
hrwwill check how stuff goes09:38
hrwfrickler: ok, cannot add work email to new account as it is assigned to old account09:48
fricklerhrw: yes, we would need to remove that from the old account first. however this is blocked by some cleanup we still need to perform on existing accounts. maybe clarkb later can tell us something about the current status09:49
fricklerhrw: actually I think I could try to remove your email address from the old account, if you want09:53
hrwif you can then it would be great09:54
hrwbut - how do you know that I am who I am?09:54
fricklerhrw: well if you are not you, then the real hrw would complain later I guess. you also need to verify your email in order to add it to the new account09:56
fricklerhrw: o.k., that seems to have worked, please try again09:59
hrwthanks. added and set as preferred09:59
hrwand will have fresh history on launchpad ;D10:00
*** rlandy is now known as rlandy|ruck11:10
*** jpena|off is now known as jpena11:27
*** dpawlik7 is now known as dpawlik11:56
opendevreviewMartin Kopec proposed opendev/system-config master: Update Interop doc
fungihrw: did something go wrong with the 2fa setup?12:55
hrwfungi: first I forgot to do 2fa, then covid came and forgot even more. then got logged out.13:12
hrwfungi: I am sorting that out.13:12
fungiahh, okay13:14
hrwfungi: in worst case I will use second account. Kolla PTL is aware of situation (the only team where I have core)13:14
opendevreviewMerged zuul/zuul-jobs master: Print version of installed podman
*** ysandeep is now known as ysandeep|dinner14:34
*** ysandeep|dinner is now known as ysandeep15:14
*** ysandeep is now known as ysandeep|out16:31
clarkbfungi: ianw  looks like certs must've renewed as we didn't get emails indicating the fix worked16:40
fungiyep, confirmed16:41
fungiwe have new "not after" dates in the certs now16:41
*** marios is now known as marios|ou16:44
*** marios|ou is now known as marios|out16:44
clarkbI'm working on fixing up my bullseye docker image builds this morning. One thing it is tripping over is installing libffi6 on bullseye. Any idea why distros don't do meta pacakges for libs like that so you can install libffi instead?16:50
clarkbfixable via bindep rules but curious why this has to be as painful as it is16:51
fungion debian it's usually because transitions need e.g. both libffi5 and libffi6 available for install at least for some short period16:53
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
fungiotherwise everything which links libffi would have to be rebuilt and uploaded at the same time as libffi6, which on its own may not sound complicated, but multiply that by thousands of similar libs and you quickly get yourself deadlocked between multiple library transitions or have to serialize them to the point where you can't finish transitions faster than you start them16:55
clarkbI see16:56
fungiit's quite common for the soname version of a lib to be embedded into the package name, as a result16:59
fungiand usually it's assumed those will only ever be consumed through declared package dependencies anyway, so users aren't inconvenienced by the package name changes17:00
fungiwhen you're installing them directly to build something against, yeah that becomes annoying17:01
opendevreviewClark Boylan proposed opendev/system-config master: Update the hound image to bullseye
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
clarkbfungi: buster-backports contains packages that are in bullseye right? This means when moving from buster to bullseye I can drop the backports specifier and keep the same version?17:11
opendevreviewClark Boylan proposed opendev/system-config master: Update matrix-eavesdrop image to bullseye
clarkbThat ^ update goes with that assumption17:12
opendevreviewClark Boylan proposed opendev/system-config master: Update refstack image to bullseye
opendevreviewClark Boylan proposed opendev/system-config master: Update the accessbot image to bullseye
opendevreviewClark Boylan proposed opendev/system-config master: Update limboria ircbot to bullseye
opendevreviewClark Boylan proposed opendev/system-config master: Install Limnoria from upstream
clarkbfungi: do you understand why says the wheel failed to build. It seems light on details. But our image builds require we build wheels for everythign on the build image and then we install the wheels on the base image. Currently failing beacuse it tries to sdist install on the17:29
clarkbbase image because the wheel failed17:29
clarkbI'm going to spin up a bullseye container locally and try t odebug17:30
fungiclarkb: yes, for a package to be in buster-backports it must be present in bullseye already (i'm almost positive that's policy)17:32
clarkbI can't reproduce this failed uwsgi build locally on top of our python builder. This is weird17:38
clarkbhuh it only failed on python 3.8 on bullseye. Which is what I'm trying to reproduce on. But maybe it was a fluke? I'll recheck and see17:40
fungii wonder where it gets a 3.8 for bullseye, the python3 provided with bullseye is 3.917:43
*** jpena is now known as jpena|off17:50
clarkbfungi: we use the "official" python docker hub images which are based on debian but then build a C python on top of that from source17:52
clarkbthe python isn't debians, the rest of the userland is17:52
clarkbheh now the 3.9 job is failing17:53
fungiahh, okay17:53
fungithat makes more sense17:53
hrwfungi, frickler: my account is restored17:54
hrwreason turned out to be more complex17:54
fungiif you can provide details, that info might be useful to others who run into the same situation trying to use 2fa17:56
opendevreviewClark Boylan proposed opendev/system-config master: Update refstack image to bullseye
hrwfungi: turned out that some 2fa settings were enabled on my account from my Canonical times (when it was mandatory for employees). When I left company, 2fa group was removed along with 2fa being disabled. I shredded access codes etc and now when they enabled 2fa for everyone turned out that some settings left so LP asked me for codes from devices I stopped using 8 years ago.17:59
clarkbok I guess that means we should prescreen people and ask if they ever had 2fa enabled previously18:00
fungithat's excellent information, thanks! i wondered if it might be something like that18:01
hrwthe good part? allows to setup 2fa without being in any 2fa groups18:01
hrwso if someone wants to enable 2fa they can do it on their own18:01
fungii didn't realize even still existed, we ported all the openids over to years ago18:01
hrwwhatever - I think it is the same backend on two urls18:02
clarkbhrw: oh if they let you opt in directly now then ya we should just push people that direction and not worry about managing this group18:02
clarkboh wait vs ubuntu one though18:03
clarkbok the bullseye python 3.9 error is the same as the bullseye python 3.8 error. It failed to build a uwsgi wheel then in the base image build we failed as we don't have the bits necessary to build the package from source (it relies on that wheel existing)18:05
fungiclarkb: yeah, looks like it's complaining that the image lacks gcc?18:06
clarkbfungi: thats the later error. Which happens because we don't put build deps in our production deploy image. The expectation is taht the builder image does all of that and produces a wheel then the production image copies that wheel and installs it. This keeps images sizes down as you don't end up with a bunch of build artifacts and dependencies18:07
fungi"Exception: you need a C compiler to build uWSGI" raised when trying to find the gcc version18:07
clarkbfungi: is the error causing ^ to happen18:07
fungioh, i didn't realize that was from a different layer operation18:08
fungii was a bit confused as to why it was bothing to invoke gcc and then later complaining it wasn't there at all18:08
clarkbIf the wheel builds as expected then we install later and don't need gcc18:08
fungihowever, the compiler error (if there was one) seems not to have been included18:08
clarkbya exactly18:09
fungicould "UserWarning: Unknown distribution option: 'descriptions'" be causing the build process to exit nonzero after compiling?18:10
clarkbhrm maybe.18:10
clarkblet me cross check against the successful builds18:10
clarkb a succesful one doesn't complain at all but I'm not sure if that is because it isn't really printing much when successful or if it is related18:14
clarkbI wonder if it is a bug in their build and doing it with all the CPUs trips a race18:16
clarkbis setuptools too old?18:19
clarkbProblem is this isn't reliably reproduceable so makes it really difficult :/18:20
jrosserit doesnt look like that has compiled as many source files as a successful run when it fails18:27
jrosserwhich would make me wonder about a missing header file, something like that18:27
fungiyeah, i wish we could get the (stderr?) output from it18:28
clarkbfungi: if you look it says complete output and gives the number of lines and it appears we have all of them?18:29
fungithat might just be stdout though?18:30
jrosserthere are approx 10 lines short of the core and it's not started the embedded plugins at all18:30
clarkbfungi: ya I suppose, Its just doing pip install uWsgi basically and that gets emitted. Maybe I should modify it to do a pip -vvv or whatever the equivalent is18:31
clarkbjrosser: interesting, maybe that does point to the buggy build process then and us hitting a race somehow18:31
jrosseri can give you a paste of a successful build, though thats kind of unhelpful for what the actual bug is18:31
clarkbI can probably get that output from a successful build too, just need to increase verbosity which seems like a good idea either way18:32
fungianother option might be to set an autohold and then check dmesg/syslog for signs of memory exhaustion et cetera18:34
fungibut hopefully increasing verbosity will get us errors in such cases18:35
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
clarkbThat ps makes pip very verbose18:37
clarkband locally that produces a complete build output so we can get it that way if we need it18:37
fungithat looks like one way of doing it, yep18:42
clarkbya easier than modifying assemble which is in another docker image :/18:43
clarkbat least since we only need it for one image right now. If we want this by default then we can update assemble directly18:43
clarkbinfra-root most changes under should be ready for review now. These update a number of our images to bullseye.18:43
clarkbIf you'd are able to review and at least +2 that would be great. I'm happy to +A one or two at a time as we are able to verify the end results18:44
fungiclarkb: i'll take a look in a moment. on an unrelated note, should we go ahead with topic:mailman-lists as well? you've +2'd them all and ianw has done so for half. or we could wait for more reviewers18:50
clarkbfungi: I think we can probably proceed carefully unless frickler  or corvus indicate an interest. Probably the iptables update is most interseting though it shouldn't affect prod18:52
clarkbbasically land a change or two and verify as we go18:52
fungiyeah, the firewall change is the first one in the series, so the rest of it is blocked on that anyway18:57
*** sshnaidm is now known as sshnaidm|afk19:06
bqianany admin can take a look at the Zuul  build error in
fungisure, taking a look now19:57
fungibqian: looks like your patch-tox-pylint job is complaining "cgcs_patch/ E0401: Unable to import 'rpm' (import-error)"20:00
bqianyes, that's what it is complaining. not sure why it is unable to import20:01
fungihave you tried to reproduce it elsewhere?20:02
bqianit runs ok in my local, and it ran ok before patch 6, the patch 6 has not changes in that file20:04
fungiwhat platform are you testing on locally?20:05
fungiwhat normally provides the "rpm" python library?20:05
fungii don't see where the test is installing it20:05
bqianI am not sure about the rpm import, as my changes don't need it20:06
bqianlet me check around20:07
fungibqian: according to this, the job wasn't run for a couple of months before your change tried to call it:
fungiodds are something happened during that span of time, probably outside of that project20:07
fungiunfortunately we no longer have logs from the last successful run since it was in october20:08
fungi(we expire build logs after a month)20:08
ianwclarkb: if you get a chance for and that tests 8-stream arm64 and fixes up mirror usage, that would be good20:09
ianwjust a double check i didn't miss something meaning it's not testing what i think it's testing20:09
clarkbianw: I should be able to look after lunch20:09
ianwbut with that, i think we can do 8-stream arm64 nodes20:09
ianwclarkb: no rush! :)20:10
ianwthere's no changes so current dib should work20:10
ianw(famous last words)20:10
clarkbThen it also looks like my pip verbosity change doesn't work on 3.7 because 3.7 bundles old pip in venv20:10
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
clarkbOnly still sort of half here, but the rtt on getting new results is long enough that I wanted to get something up while I finish digesting20:22
clarkbfungi: bqian I suspect you need to run that job on a centos or fedora node20:24
fungiright, i'm wondering if what happened recently is that some other change switched the job to use ubuntu20:25
clarkbor maybe pylint started cehcking imports by importing them?20:25
fungialternatively, there are python-rpm and python3-rpm packages available on bionic20:26
fungiclarkb: the error indicates the import is embedded in another script in that repo20:26
fungicgcs_patch/ (line 17)20:26
fungioh, started checking imports is what you said20:26
fungiyeah, could be a new pylint version20:26
fungior something changed the exclusions list for pylint20:27
bqianthanks fungi20:39
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
fungi#status log Jitsi-Meet services on are back in service again following an upgrade to the most recent image builds21:05
opendevstatusfungi: finished logging21:05
clarkbfungi: responded to your question on note a similar reason applies for refstack21:08
clarkbof course as soon as I get a working dockerfile up with extra pip verbosity all of the builds succeed :)21:19
clarkbI've rechecked. It really wouldn't surprise me if this is a build system timing issue if changing verbosity changes behavior and it only fails every once in a while21:20
clarkbhound is probably a good next bullseye update: if you have time to be a second reviewer on that ianw21:30
clarkbianw: also is the bottom of fungi's improve mailman testing stack which includes firewall updates in testing if you have time for that22:04
clarkbI think we should carefully land that one22:04
clarkbMeeting agenda has been updated. Please edit to add anything I missed or let me know and I'll add it in then probably send out in an hour or so22:07
opendevreviewMerged opendev/system-config master: Update the hound image to bullseye
ianwthat stack lgtm, only question is if bits would be better in a test playbook22:13
clarkbya I thought about that too, but in this case it seemed fine since it is tied to the specific check in the test case. But I could go either way22:14
clarkbjust waiting for the hourly runs to finish then hound should update22:17
clarkbI'm keeping an eye on it22:18
fungiright, if there's a good way for me to insert a restart playbook between the production deployment playbook and the testinfra run, i'm happy to do it that way22:18
clarkbfungi: ya you can specify a test playbook like gerrit uses22:18
clarkbfungi: look in zuul.d/system-config-run.yaml and the system-config-run-review-base job22:19
clarkbit lists the prod playbooks then a test specific playbook that runs after the prod playbooks22:19
fungithanks, i'll try to get it working tonight22:19
opendevreviewMerged openstack/diskimage-builder master: Test 8-stream aarch64 build
clarkbhound is restarting now22:38
clarkbseems to be doing what I expect it to. It is setting up searchers for all the repos22:38
clarkband it is up, queries work. I think it is happy22:41
clarkbI cannot get the uwsgi builds to fail with extra verbosity. I'll try removing the verbosity and see if they fail again. If so maybe we leave the verbosity in there as a bug fix <_<22:42
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
clarkbianw: is probably a good next step for docker images if you want to look at another one. The change there bumps gerritbot to bullseye then the child sets the same uid and gid that we set on the service now22:45
ianwclarkb: lg, is there a depends-on for it?22:56
ianwin system-config i mean22:56
clarkbianw: no we run the eavesdrop playbook hourly for this reason. A number of eavesdrop things happen without easy cross tenant triggering22:59
clarkbsimilar to why we run zuul and nodepool that way 22:59
ianwnp, it all looks good to me, yeah we just want to watch it.  i did think the eavesdrop test started this and connected to opendev-sandbox, maybe that's something else23:00
ianwdoes anyone have a problem with the openeuler nodes in ?23:01
ianwit's been sitting for a while.  space is tight, but since we've accepted this in dib, etc. it seems like this has been the ultimate goal23:02
ianwon arm64 if we add 8-stream, 9-stream and this we may be very very tight23:02
ianw /dev/mapper/main-main  787G  526G  262G  67% /opt23:03
clarkbI think we can start moving forward on removing gentoo and opensuse tumblweed which will free up space (mostly tumblweed not sure we ever mirror'd gentoo stuff)23:05
clarkband then we can followup with opensuse leap 15 removal and centos-8 removals23:05
clarkboh you don't mean mirrors you mean actual builder room23:06
clarkbsorry I misinterpreted23:06
clarkbI think only centos-8 will be removed on the arm builder side and that is the last thing we said we would remove23:06
clarkbianw: No objcetions from me but should we add it to more clouds or are you wanting to do a smaller number to start?23:07
clarkboh i see a comment says one provider to start ok +2 from me23:08
clarkbI didn't approve it as I'm not sure I'll be around long enough to monitor the resulting image builds and ensure we don't run out of disk23:08
ianwyeah i'm not really wanting to do anything :)  but probably not a bad idea to debug in one env only first23:08
ianwcorvus: is one to update our logging to use the multilineformatter you put in23:09
ianwthe other thought i had was dropping the formatter section, but i wasn't sure if it would fall back to the default correctly if reading a logging config file23:10
clarkbmeeting agenda sent now as promised about an hour ago :)23:11
ianwi'm hoping to get a spec on credentials from zuul in shape before then, not quite done23:11
clarkbthanks, it is on the agenda for discussion either way23:12
clarkbok the uwsgi base images don't build when I drop the verbose flag23:12
clarkbI'm going to propose it with the verbose flag23:12
opendevreviewClark Boylan proposed opendev/system-config master: Properly build bullseye uwsgi-base docker images
clarkbAnd yes that is crazy, but meh23:14
clarkbI've approved the gerritbot bullseye update but will do the user update separately just in case23:19
opendevreviewMerged opendev/gerritbot master: Update docker image to bullseye
ianwclarkb: hrm ... really looks like there was no gcc installed for amd64, but it was for arm64?23:28
ianwthis might be the case as arm64 were are more attuned to having these dependencies instead of relying on wheels23:29
ianwthat was from ps3 @ ... is that representitive of the problem?23:29
clarkbianw: that error is misleading23:32
clarkbthere isn't a gcc installed beacuse that is on the python-base image where we copy over the wheels and install the package. The problem is earlier when we try to build the wheel it fails so we don't have a wheel and it tries to install and fails due to no gcc23:33
clarkbThere is something up with the uWSGI build that I cannot reproduce locally and adding verbosity also seems to make go away23:33
ianwahhh, ok, more like 2021-12-13 17:22:15.939811 in ?23:38
clarkbyup that happens then we don't get a wheel so it looks for gcc later when we don't want it to have gcc (its done in an effort ot keep image sizes down)23:39
ianwyeah, ok that makes sense23:41
fungiclarkb: what are the odds we tripped over a uwsgi release which only had an sdist and no manylinux wheels matching that interpeter abi?23:45
fungii guess slim, last release was in early october23:45
clarkbya the reason we go through this trouble is uWSGI needs a proper build23:48
ianw2021-12-13 23:21:15.763065 | ubuntu-focal | #13 31.19   [thread 7][gcc -pthread] lib/linux_ns.o23:50
ianw2021-12-13 23:21:15.864407 | ubuntu-focal | #13 31.23   [thread 5][gcc -pthread] core/yaml.o23:50
ianw2021-12-13 23:21:15.864455 | ubuntu-focal | #13 31.23   [thread 0][gcc -pthread] core/ssl.o23:50
ianwthese seem to be 3 things the successful verbose build creates that don't appear on the failed build 23:50
ianwi should say 3 suspicious things23:51
clarkbwe don't explicitly add libyaml to the bindep. But libssl-dev is there. Linux binding stuff could also be cleaned up maybe in some cases beacuse we don't hve a kernel on the image?23:52
clarkb doesn't say anything about those extra bits though23:53
fungithis was working on buster but isn't on bullseye, right? or is now working at random on bullseye?23:55
clarkbya I haven't seen this failuer on buster. It works randomly on bullseye and fails randomly on bullseye. I'ev done a number of local bullseye builds and havne't been able to reproduce. It always seems to work here23:55
clarkbthis is why i suspect a timing issue and I suspect doing more IO for pip verbosity improves that23:55
clarkbinitially all I wanted was the pip verbosity to expose what was failing better. But now it seems to just work with that flag :/23:56
fungiand heisenbugs out when you add the extra verbosity so we have no clue what it's failing on, right23:56
clarkbThis is like the 5th rebuild in a row where all three python versions on bullseye built with verbosity set23:56
clarkbbut when I remove verbosity at least one fails in every buildset23:56

Generated by 2.17.2 by Marius Gedminas - find it at!