Wednesday, 2021-10-20

fungiclarkb: yes, i think we need new xenial images with working bindep preinstalled, right?00:02
fungii'm just pre-approving it so it can land with a recheck later00:02
Clark[m]Ah ok. I was wondering if we wanted to force merge and approve it but I think either way we need new images so waiting makes sense00:03
fungior we can shove it past zuul with admin powers and tag a new point release00:03
fungiyeah, that00:03
fungii'm good with either solution00:04
Clark[m]Ya now that I realize we are waiting for new images either way waiting to land it properly makes sense00:04
fungithe image fixing would need 814677 landed, right?00:04
opendevreviewMerged openstack/project-config master: Add ubuntu-bionic-32GB
fungido we want to revise it for your comment about using an inequality instead of an exact version?00:06
fungiotherwise i'll approve ow00:07
fungiwent ahead and approved it00:12
fungiwe can always loosen that later if we want00:12
fungiwe'll eventually be dropping xenial anyway eventually00:13
fungidouble-eventually consistent00:13
Clark[m]Ya I think approving it as is makes sense for now00:17
opendevreviewMerged openstack/project-config master: infra-package-needs: install latest pip
ianwi can trigger some xenial builds or we can just wait00:22
fungiwhichever you prefer, but one of those options definitely sounds like less work00:39
ianwfwiw seems happy again02:44
opendevreviewMerged openstack/diskimage-builder master: Remove py35 tox jobs
opendevreviewIan Wienand proposed zuul/zuul-jobs master: ensure-docker: remove Debian Stretch testing
opendevreviewMerged opendev/base-jobs master: Remove debian-stable nodeset
opendevreviewIan Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing
*** pojadhav|out is now known as pojadhav|ruck04:04
*** ykarel_ is now known as ykarel04:40
opendevreviewMerged openstack/project-config master: Remove fedora-32 disk image config
*** ysandeep|out is now known as ysandeep05:08
chandankumarianw: Good morning, please have a look at this when free, thanks :-)05:12
opendevreviewSandeep Yadav proposed zuul/zuul-jobs master: multi-node-bridge: repos to install ovs in C9
opendevreviewMerged openstack/diskimage-builder master: epel: match replacement better
opendevreviewMerged openstack/diskimage-builder master: Revert "Allowing ubuntu element use local image"
opendevreviewMerged openstack/diskimage-builder master: ubuntu-systemd-container: deprecate and remove jobs
opendevreviewMerged openstack/diskimage-builder master: ubuntu: add Focal test
opendevreviewMerged openstack/diskimage-builder master: functests: drop apt-sources
opendevreviewMerged openstack/diskimage-builder master: centos7 : drop functional testing
opendevreviewMerged openstack/diskimage-builder master: functests: drop minimal tests in the gate
opendevreviewMerged openstack/diskimage-builder master: Remove extras job, put gentoo job in gate
opendevreviewMerged openstack/diskimage-builder master: Simplify functests job
*** ysandeep is now known as ysandeep|afk06:34
*** ysandeep|afk is now known as ysandeep|trng07:03
priteauGood morning. Is Zuul struggling today? has been queue for 45 minutes without starting jobs.07:19
opendevreviewMerged openstack/diskimage-builder master: Run functional tests on Debian Bullseye
fricklerpriteau: I think that patch was just unlucky to come in right after the daily bunch of periodic jobs fired07:29
*** jpena|off is now known as jpena07:29
fricklerinfra-root: looking at the grafana data, maybe we could move the periodic trigger a bit earlier to avoid clashing with EU work day start so much? seems we mostly have a pretty idle period from roughly 0-6 UTC07:32
priteauOK, just need to be patient I guess07:32
opendevreviewDr. Jens Harbott proposed opendev/project-config master: Move the daily periodic trigger earlier
*** ykarel is now known as ykarel|lunch08:00
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB
*** iurygregory_ is now known as iurygregory09:00
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: [wip] Add support for CentOS Stream 9 in DIB
louroto/ we (openstack charmers) would love to get this one landed, thanks!
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky
*** ykarel|lunch is now known as ykarel10:15
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky
opendevreviewMichal Nasiadka proposed openstack/diskimage-builder master: Add new element rocky
*** ysandeep|trng is now known as ysandeep|afk10:19
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB
*** dviroel|rover|afk is now known as dviroel|rover11:18
*** jpena is now known as jpena|lunch11:25
*** ysandeep|afk is now known as ysandeep12:21
*** jpena|lunch is now known as jpena12:24
*** pojadhav|ruck is now known as pojadhav|brb12:32
*** pojadhav|brb is now known as pojadhav|ruck12:43
opendevreviewMerged zuul/zuul-jobs master: multi-node-bridge: repos to install ovs in C9
fungifrickler: i have no objection to moving the trigger time for the various periodic pipelines, though also i'll reopen the discussion with the openstack stable maintainers about removing periodic jobs if there's no hope of them succeeding12:51
fricklerfungi: I already proposed a patch earlier. but I agree removing known-failing jobs should happen in parallel, too12:52
fungiahh, i see now, you pushed up 81472112:54
*** timburke_ is now known as timburke12:58
clarkbI too have no objection with better tuning periodic trigger times. ++ to cleaning up the queues though13:51
fungiyeah, elodilles wanted to keep those failing stable periodic jobs around to see when they get fixed, but i still get ~110-120 failure reports in my inbox every morning so i don't think the number is shrinking13:52
elodillesit shrinked, but unfortunately just a very bit o:)14:19
elodillesand worth to mention that most of the failing periodic jobs came from oslo, rocky and older projects, that is currently about to move to EOL14:20
elodillesat least the team sent a mail to ML about that14:21
elodillesthis was the case so far... but for example today I see double amount of failing periodics than the "usual" 115 :S14:22
opendevreviewDouglas Viroel proposed zuul/zuul-jobs master: Add FIPS enable multinode job definition
opendevreviewMerged opendev/project-config master: Move the daily periodic trigger earlier
clarkbfrickler: ^ did you want to write a corresponding change for openstack/project-config too? or should I?14:25
*** andrewbonney_ is now known as andrewbonney14:26
fricklerclarkb: on it14:31
opendevreviewDr. Jens Harbott proposed openstack/project-config master: Move the daily periodic trigger earlier
opendevreviewMerged openstack/diskimage-builder master: Update centos element for 9-stream
fungielodilles: the additional failures today are that basically any jobs run on ubuntu-xenial failed because of a problem with how we were installing the latest bindep release from monday into the images when we built them14:54
opendevreviewMerged opendev/bindep master: Add old python packaging pin
opendevreviewAlfredo Moralejo proposed openstack/diskimage-builder master: Add support for CentOS Stream 9 in DIB
elodillesfungi: indeed, I saw that rocky, queens and pike have the new failures, those are on xenial15:07
elodillesfungi: I've checked now the failure ( )15:08
prometheanfireI suppose making a new bug for gertty traceback (git master) when trying to pull up a diff? or is there an irc channel?15:09
fungiprometheanfire: are you using latest master of gertty? there are fixes which address problems like you're describing15:14
fungiinfra-root: proposing to tag opendev/bindep commit 0bd79b1d5dd94570f8e3629b991642e919acdb94 as 2.10.115:14
clarkbfungi: checking15:14
fungii have the tag created locally, just waiting to push it to gerrit15:15
clarkb0bd79b1d5dd94570f8e3629b991642e919acdb94 as 2.10.1 lgtm15:15
fungithanks, i'll go ahead and push since this is a fairly important fix for people with old pip versions15:16
fungiand done15:16
fungii'll send a release announcement once it's up on pypi15:16
elodillesthanks for the fix and the release!15:21
prometheanfirefungi: ya, just installed15:28
*** ykarel is now known as ykarel|away15:31
*** marios is now known as marios|out15:41
fungiprometheanfire: are you seeing the crash on master?15:46
fungiprometheanfire: some specific change you can reproduce it on? i'll see if i can come up with a patch15:49
prometheanfiresure   repository:     
prometheanfire   at the commit:            559bcfdb33769dd1b9db499daf7443dae06d09a815:49
prometheanfirethat's the gertty version installed15:49
fungii mean what change are you pulling up in gertty15:50
prometheanfirewhen trying to view the diff15:50
prometheanfireI know, wanted to get you the full info :P15:51
fungithanks, seeing if i can reproduce15:51
fungistrangely i can look at the diff view for that change with no problem. pbr freeze reports i'm using 559bcfd as well15:53
fungimaybe it's because i'm using unified diff mode15:54
clarkbpossibly also a dep issue?15:54
fungiprometheanfire: can you see if it still happens with "diff-view: unified" in your .gertty.yaml?15:55
fungii'll try switching mine to the default (side-by-side)15:55
fungidoesn't crash for me with sxs mode either15:56
prometheanfireupdating gitpython seemed to help15:59
fungii take it that appeared in the traceback?15:59
prometheanfirefrom 3.1.20 to 3.1.2415:59
fungii've got GitPython==3.1.2415:59
prometheanfireyou could try with 3.1.20 if you care to verify, but 3.1.20 works, not sure why it failed all of a sudden, been a couple of months fine with 3.1.2016:00
prometheanfire3.1.20^t{xpak}(03:08:18 PM 08/27/2021)16:00
fungiprometheanfire: could be related to our recent gerrit upgrade "somehow" i suppose16:01
prometheanfireya, wfm16:01
fungithis is what `pbr freeze` reports for my current gertty venv (running under python 3.10):
fungii'll try to force GitPython down to 3.1.2016:03
*** pojadhav is now known as pojadhav|ruck16:12
*** pojadhav|ruck is now known as pojadhav|out16:12
fungiugh, internet provider seems to have decided 16:00 utc on a wednesday is the ideal time to take everything offline16:13
fungiafter all, who'd be home in the middle of a weekday?16:13
fungianyway, i'm limping along via cellular modem for the time being16:14
fungiprometheanfire: downgrading to GitPython==3.1.20 doesn't seem to have allowed me to reproduce the crash either16:15
fungidid you happen to upgrade anything else along with gitpython?16:16
* prometheanfire shrugs16:16
fungiand do you still have your earlier traceback?16:16
fungii wonder if i need to be on python 3.9 instead of 3.10 to reproduce that, it's being raised in the stdlib typing module16:20
prometheanfirecould be16:21
fungii'll recreate my venv with 3.9, just a sec16:23
fungiunfortunately somewhat slower over 3g tin-cans-n-string16:25
clarkbfungi: your island hasn't upgraded to fake 5g yet?16:26
fungiTypeError: Protocols cannot be instantiated16:26
fungiprometheanfire: so seems to be python 3.9 + gitpython 3.1.2016:26
fungiupgrading to latest gitpython fixed it for me too16:28
fungiprometheanfire: looks like that's
fungiseems python 3.9.6 was fine but 3.9.7 caused it to break, so maybe you upgraded to 3.9.7 recently?16:30
fungithe workaround was included in gitpython 3.1.2216:31
*** jpena is now known as jpena|off16:31
fungianyway, i'm switching my gertty back off this ancient 3.9 interpreter back to a nice clean 3.10 ;)16:31
*** ysandeep is now known as ysandeep|away16:41
opendevreviewClark Boylan proposed opendev/system-config master: Update zuul restart documentation
clarkbfrickler: ^ fyi16:50
opendevreviewJeremy Stanley proposed opendev/bindep master: DNM: checking that jobs still run
fungiclarkb: not sure if you saw, but 813301 and 813303 update our specs template based on your comments for the mailman3 spec17:06
clarkbfungi: I hadn't I'll take a look once I've got this next change about debugging zuul behaviors pushed17:07
fungino worries, just making sure you knew they were there17:11
prometheanfirefungi: nice17:12
prometheanfirefungi: yep, 3.9.7_p1(3.9)^t{xpak}(07:33:10 PM 10/15/2021)17:12
opendevreviewClark Boylan proposed opendev/system-config master: Add zuul debugging documentation
clarkbfrickler: ^ and now a second change to talk about debugging process a little bit17:13
clarkbthat is a bit more hand wavy but hopefully gives enough hints to what I've found useful that others can make use of it17:13
clarkbfungi: I went ahead and approved the spec template updates, thanks for those17:16
fungisure, i just didn't want to forget to push them up17:20
opendevreviewMerged opendev/infra-specs master: Add backups section to template
opendevreviewMerged opendev/infra-specs master: Simplify assignees section of template
clarkbfungi your dnm bindep change passed its tox-py35 build. I think we're good there now17:25
clarkblooks like the dib changes all landed after the py35 job was removed so nothing to recheck there17:34
clarkbMy doc changes appear to have rendered well enough. I'm not sure I like italics for inline quoted stuff. But I think it is a theme change to change that17:42
clarkbfungi: I checked on because it failed a bunch of bindep stuff. But the issue is the lack of the newline in that file17:56
clarkbI guess parsely or whatever parser bnidep uses doesn't like that17:56
clarkbI wonder what editor people are using that doesn't add the newline17:59
fungii see it pretty often, yeah18:08
fungino idea what sane text editor would do that though18:09
clarkbfungi: I think your docs snippet on decoding zuul secrets is no longer valid due to the up to date secrets living in zk18:09
clarkbI'm not yet sure what the best way to extract the info is though. Maybe export-keys then something sort of python -m json script or jq usage?18:10
fungireuse our key backups?18:10
clarkbya that is an option. The only thing is its a giant json file with everython on one line iirc. So you'll need a smart tool to pull out the info you want18:10
clarkbjq should be able to do it I guess18:11
fungias long as the project wasn't added in the past 24 hours, that's probably easier than trying to extract them from zk18:11
fungiyeah i'd probably use q18:11
clarkband then you can always manually run export-keys too to update the file18:11
clarkbnow the next problem becomes figuring out the jq incantation to make this work. its syntax does not come very naturrally18:12
fungiit's sql-like right?18:12
fungior am i thinking of a different utility?18:12
clarkbI don't think so. You use a python addressing like syntax but with weird quoting iirc18:13
clarkbjq ´.foo?´ -> returns foo if present in the dict and null otherwise18:13
fungiahh, i'm thinking of something else just called q which lets you query json files as if they're sql databases, i think, but the name makes it very hard to find18:14
clarkb{"keys": {"fullkeypath": {"keys": [{"private_key": "key data here"}]}}} <- is the datastructure we are trying to extra private_key from18:17
clarkbok I think I have it18:20
fungiokay, it was i was misremembering, it's for querying sqlite not json18:20
fungiso yeah jq is likely our best bet18:20
clarkbhrm now running into issues with /'s in the dict key names that I have to figure out18:24
fungibecause json is web-scale?18:25
clarkbwell because shell quoting and weirdness. This is always what I run into with jq when I try to use it. But I figured it out18:25
opendevreviewClark Boylan proposed opendev/system-config master: Update zuul secrets decryption docs
clarkbI've not tested ^ on anything more than a local mockup of secret key backups18:34
clarkbfungi: ^ I think you understand all of this better than I do so maybe carefulyl review it and give it a go and feel free to update with new patchsets?18:34
clarkbcorvus: ^ also I'm not sure if that sort of thing might want to live in zuul docs?18:39
clarkbcorvus: maybe it would be better for me to psuh something there then opendev can link to it?18:40
fungii'm still slowed a bit by a broadband outage but will have a look shortly19:01
fungidoesn't help when t-mobile decides to just stop talking to the modem for 5 minutes at a stretch19:16
fungito my backup cellular modem i mean19:17
fungiwhee, a zuul-jobs change approved almost two hours ago is still waiting on a f34 node in the gate. might be time to switch some projects back to testing with f33 until f34 is actually bootable most places20:03
opendevreviewMerged zuul/zuul-jobs master: build-python-release: Add flag for whether to build a wheel or not
clarkbI'm looking at making bindep handle no trailing newline and the trivial method would be to add the newline if it isn't there already. But I really want to address this in the grammar but I just end up with parsing loops20:46
clarkbanyway I'll have to look at this more closely after the dentist20:46
fungiyay, my internet finally came back21:33
ianwfungi: i don't think we ever had f33 nodes, and f32 is gone -- so ...21:38
ianw9-stream is closer21:38
ianwsomething that might be good, and i just notice dthis looking at that rocky patch too, if we could consistently fix our grub to make sure we get serial output21:47
fungiianw: aha, i hadn't been paying close enough attention to realize f33 never materialized21:53
fungitimburke_: 795419 merged at 20:32 utc, so time to work out next steps for getting a release uploaded (whether we can re-enqueue the most recent tag or a new one is needed will depend on what, if anything, may need to merge to that repo to use the new flag)22:00
fungihappy to talk through options next time you're free22:00
timburke_fungi, thanks for the ping! i'll recheck and go from there22:02
fungigreat, let me know if you need anything at all22:03
timburke_i don't *think* we need to do any more work on existing tags -- i see tarballs for the current release at both and
timburke_for better or worse, that library changes so infrequently that we don't really have need of another release any time soon, i think22:06
timburke_(though getting manylinux wheels building might be a good reason, if i can get that working ;-)22:07
fungitimburke_: cool, i can try reenqueuing the most recent tag to get it uploaded to pypi, if that's good with you22:14
fungilooks like that's 1.6.022:15
timburke_shouldn't be needed -- looks like it's already there22:15
fungiahh, okay22:15
fungiyeah, i agree it's already there22:16
timburke_this was all noticed because of an attempt to update the README -- nothing major, but enough to make the test-release-openstack job run22:16
fungiso i guess make a new release request to tag the next version you want, looks like you've landed 16 changes since 1.6.022:17
fungianyway, if you need more help with it, don't hesitate to let us know22:18
ianwfungi: tox-py35 still fails, but now in a different way it seems22:28
ianwit's decided to build a wheel for cffi and failed22:30
ianwthis might related back to the fact that the wheel mirrors are unhappy22:30
ianwi guess that's what i'm doing now22:30
fungiyeah, presumably no cffi wheel on pypi for that platform/interpreter22:31
fungicffi!=1.11.3,>=1.8 so it's settled on 1.15.022:32
fungiwhich is the most recent cffi release from a week ago22:32
fungiand indeed, only wheels for python >=3.622:33
fungibut hey, they have wheels for python 3.10 at least, if you need those22:33
ianwcp: cannot create regular file '/afs/': Disk quota exceeded22:34
ianwhere's problem 1 / X22:34
ianw(X to be determined ...)22:34
fungiZeroDivisionError: division by zero22:35
ianwcp: cannot create regular file '/afs/': Disk quota exceeded22:35
ianwarm64 same issue22:35
fungisame volume, yeah?22:35
ianwthe xenial wheels are failing because ... bindep :
fungi suggests we still have space on the mirror.centos volume22:36
ianwi think they have their own volumes22:37
fungioh, right, this is the wheels22:37
* fungi scrolls down22:37
fungioh, yep22:37
ianwthe quota is 5mb22:38
fungislap full22:38
ianwthat seems an oversight ...22:38
fungi5gb is the default i think? so we probably forgot to set a quota on some of these22:38
fungifocal x86_64 looks similarly problematic, but i guess has no job yet22:38
ianwyeah not 5gb, 5mb 22:39
fungithe other wheel volumes have 100gb quota22:39
fungier, yeah, i meant 5mb, but it's been so long since i needed to ytpe mb that my fingers just corrected it22:39
* fungi thinks back to that 5mb hardcard he installed in his computer several decades ago, before hard drives were in vogue22:40
fungiit was like having 10 floppy disks permanently inserted22:41
ianw100gb seems too much but i just upped both centos-8-stream's to taht22:41
fungithankfully, overcommits are just assumed with afs22:42
ianwthey are all a 100gb now22:42
fungii'm willing to bet it was me that forgot to set quotas when adding them22:43
fungiand i even remember adding that reminder to the docs so i'd stop forgetting22:43
ianwi feel like this is a "me" problem but happy to share blame :)22:43
ianwnow, the problem is that the release job won't release unless all the wheel builds complete22:44
fungithat also seems overly optimistic22:44
ianwso we have the question -- will the broken bindep env stop the wheel build working, stopping the wheel release22:44
fungiwill it not try to upgrade bindep?22:45
ianwyes this is an artifact of the way the jobs are structured in zuul, where the release job depends on the build jobs22:45
fungi2.10.1 should force python 3.5 venvs to get a working version of the packaging module at least22:45
fungiwhat platform(s) do we still have a broken bindep env on?22:46
fungithe xenial images should be okay now22:47
ianwyeah, i think it's just that the periodic jobs haven't run since 22:48
fungithat publish-wheel-cache-ubuntu-xenial failure you linked was before the new image got built22:48
ianw is a different issue.  that's actually tox not having libffi-devel installed22:48
ianwreally, as a generic thing in zuul-jobs, i guess that it should22:49
fungiwe realized around 14:00 utc that we could just delete the most recent image and fall back to the working one from the day before until the new one finished building and uploading, so anything which ran on ubuntu-xenial nodes after that no longer exhibited the issue22:49
ianwahh, yes that's a good idea22:50
ianwactually ... the problem is *glean* should be specifying this in it's bindep22:50
ianwglean doesn't even have a bindep22:51
ianwi think we're up to problem 4? :)22:51
fungiyeah, anything which uses cffi should strongly recommend libffi-dev/libffi-devel and a c compiler, because at some point it will be installed onto a platform which, at least temporarily, has no available cffi binary wheel22:52
fungias is the case with latest cffi and python 3.522:53
ianwi guess it doesn't really -- it doesn't have any requirements -- but it's test-requirements.txt pulling this in22:53
ianwso not required for glean runtime, but yes to run glean tox22:53
fungiof course, once you clear that hurdle, problem 5 is likely that latest cffi has dropped python 3.5 support and won't actually be buildable with it anyway22:53
*** dviroel|rover is now known as dviroel|rover|afk22:53
ianwyeah, problem 6 is if i should try to kick off the periodic wheel build jobs manually22:54
fungidoing an end run around problem 5, the cffi package only includes trove classifiers for python 3.6 and later, though curiously doesn't set requires_python metadata22:55
ianwwhile we still have xenial hanging around, we really do what glean to be py35 clean22:56
fungithankfully, the cffi changelog doesn't mention dropping python 3.5 support, so maybe it'll still work if you can get the latest version to build and install22:56
ianwdo we have bindep-test.txt.  i feel like we discussed this ...22:57
ianwsince the point of glean is to be the zero-dependency network init tool, it feels a bit odd to flag as requiring things in bindep it doesn't actually for runtime22:58
Clark[m]You can tag things test in the normal bindep file22:58
fungitangential, but i'd never heard of heptapod until reading the cffi changelog just now, since that's where they moved their source code:
fungiianw: yeah, most of our jobs install anything in the default bindep profile as well as the test profile. just add [test] after the package name22:58
opendevreviewIan Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing
ianw^ that might get things going23:02
fungiand looking around for information on heptapod led me to discover which we might want to look closer at23:04
fungimaybe not for us, but seems to be trying to bridge things like defect and task tracking between different code hosting platforms23:07
fungiianw: you have a typo there, noted inline23:08
fungioh, hah, loïc dachary is one of the folks who started fedeproxy23:11
opendevreviewIan Wienand proposed opendev/glean master: Remove debian-stable, add focal/bullseye testing
funginow i'm thinking i already knew about it and had since forgotten23:12
*** dviroel|rover|afk is now known as dviroel|rover|out23:16
opendevreviewMerged zuul/zuul-jobs master: Add FIPS enable multinode job definition

Generated by 2.17.2 by Marius Gedminas - find it at!