Tuesday, 2020-01-21

*** jamesmcarthur has quit IRC00:12
*** jamesmcarthur has joined #openstack-infra00:12
*** zxiiro has quit IRC00:13
*** jamesmcarthur has quit IRC00:18
*** jamesmcarthur has joined #openstack-infra00:24
*** igordc has quit IRC00:25
*** igordc has joined #openstack-infra00:25
openstackgerritTristan Cacqueray proposed zuul/zuul master: docs: remove generated toc from the main index  https://review.opendev.org/70346800:28
*** igordc has quit IRC00:49
*** ricolin has joined #openstack-infra00:55
*** rlandy is now known as rlandy|bbl00:55
*** artom has quit IRC01:00
ianwclarkb: LGTM, thanks!01:01
*** jamesmcarthur has quit IRC01:04
*** jamesmcarthur has joined #openstack-infra01:04
*** jamesmcarthur has quit IRC01:06
*** jamesmcarthur has joined #openstack-infra01:06
*** yamamoto has joined #openstack-infra01:15
*** jistr has quit IRC01:17
*** jistr has joined #openstack-infra01:19
ricolinfungi, can you help to create multi-arch-sig-core gerrit group, thanks!:)01:31
*** lseki has quit IRC01:40
*** rfolco has quit IRC01:41
*** Lucas_Gray has quit IRC01:49
clarkbricolin: our tooling automatically creates new groups when they are added in acl files01:56
clarkbricolin: that means you should update openstack/project-config/gerrit/acls as appropriate and the group will be created01:56
*** jamesmcarthur has quit IRC01:56
*** jamesmcarthur has joined #openstack-infra01:57
ricolinclarkb, so I guess I should ask to put me in that group right?:) (if this is correct config https://review.opendev.org/#/c/703323/3/gerrit/acls/openstack/multi-arch-sig.config )01:59
*** jamesmcarthur has quit IRC02:02
clarkbricolin: done02:03
ricolinclarkb, awesome!02:05
ricolinthank you02:05
*** xinranwang has joined #openstack-infra02:06
*** yamamoto has quit IRC02:19
*** jamesmcarthur has joined #openstack-infra02:24
*** goldyfruit has quit IRC02:30
*** roman_g has quit IRC02:34
*** ociuhandu has joined #openstack-infra02:47
*** ociuhandu has quit IRC02:51
*** jamesmcarthur has quit IRC03:04
*** jamesmcarthur has joined #openstack-infra03:05
*** dklyle has joined #openstack-infra03:07
*** apetrich has quit IRC03:10
*** jamesmcarthur has quit IRC03:11
*** yamamoto has joined #openstack-infra03:14
openstackgerritIan Wienand proposed opendev/system-config master: Remove unused linaro credentials  https://review.opendev.org/70353403:16
openstackgerritIan Wienand proposed opendev/system-config master: Add Linaro US cloud  https://review.opendev.org/70353503:16
*** jamesmcarthur has joined #openstack-infra03:30
*** jamesmcarthur has quit IRC03:33
*** hwoarang has quit IRC03:39
*** rlandy|bbl has quit IRC03:39
*** hwoarang has joined #openstack-infra03:40
openstackgerritIan Wienand proposed opendev/system-config master: Add Linaro US cloud  https://review.opendev.org/70353503:45
*** hongbin has joined #openstack-infra04:03
*** tetsuro has quit IRC04:18
*** tetsuro has joined #openstack-infra04:19
*** tetsuro has quit IRC04:23
*** goldyfruit has joined #openstack-infra04:34
*** hongbin has quit IRC04:39
*** jamesmcarthur has joined #openstack-infra04:42
*** udesale has joined #openstack-infra04:44
*** udesale has quit IRC04:44
*** udesale has joined #openstack-infra04:44
openstackgerritIan Wienand proposed opendev/system-config master: Add Linaro US cloud  https://review.opendev.org/70353504:55
*** tetsuro has joined #openstack-infra05:03
*** ykarel|away is now known as ykarel05:23
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-infra05:34
*** udesale_ has joined #openstack-infra05:34
*** jamesmcarthur has quit IRC05:35
*** udesale has quit IRC05:37
*** jamesmcarthur has joined #openstack-infra05:37
*** jamesmcarthur has quit IRC05:43
*** raukadah is now known as chandankumar05:45
openstackgerritMerged openstack/diskimage-builder master: dib-lint: test elements have README.rst file  https://review.opendev.org/17783205:50
*** surpatil has joined #openstack-infra06:03
*** SurajPatil has joined #openstack-infra06:04
*** yolanda has quit IRC06:04
*** jamesmcarthur has joined #openstack-infra06:06
*** lpetrut has joined #openstack-infra06:08
*** lpetrut has quit IRC06:09
*** lpetrut has joined #openstack-infra06:10
*** jamesmcarthur has quit IRC06:13
openstackgerritSimon Westphahl proposed zuul/nodepool master: Handle event id in node requests  https://review.opendev.org/70340606:23
openstackgerritSimon Westphahl proposed zuul/nodepool master: Centralize logging adapters  https://review.opendev.org/70340706:23
*** adriant has quit IRC06:40
*** adriant has joined #openstack-infra06:41
*** dchen has quit IRC06:43
*** dchen has joined #openstack-infra06:44
*** lpetrut has quit IRC06:49
AJaegerconfig-core, please review https://review.opendev.org/69809106:58
*** icey has joined #openstack-infra07:04
*** lmiccini has joined #openstack-infra07:04
*** jamesmcarthur has joined #openstack-infra07:09
*** jamesmcarthur has quit IRC07:14
openstackgerritAndreas Jaeger proposed opendev/system-config master: Don't publish doctrees when building docs  https://review.opendev.org/70354407:24
*** jtomasek has joined #openstack-infra07:24
*** yolanda has joined #openstack-infra07:25
*** roman_g has joined #openstack-infra07:30
*** ociuhandu has joined #openstack-infra07:30
*** yolanda has quit IRC07:33
*** yolanda has joined #openstack-infra07:34
*** ociuhandu has quit IRC07:35
*** ykarel is now known as ykarel|lunch07:36
*** lpetrut has joined #openstack-infra07:38
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: fetch-sphinx: Exclude doctrees directory  https://review.opendev.org/70354707:39
*** yolanda has quit IRC07:48
openstackgerritSimon Westphahl proposed zuul/nodepool master: Pass node request handler to launcher base class  https://review.opendev.org/70354907:50
*** pgaxatte has joined #openstack-infra08:00
*** florianf has joined #openstack-infra08:02
*** slaweq has joined #openstack-infra08:03
*** bnemec has joined #openstack-infra08:06
*** jamesmcarthur has joined #openstack-infra08:10
*** tkajinam has quit IRC08:10
*** iurygregory has joined #openstack-infra08:11
*** jamesmcarthur has quit IRC08:14
*** lmiccini has quit IRC08:17
*** tesseract has joined #openstack-infra08:20
*** yolanda has joined #openstack-infra08:21
*** lmiccini has joined #openstack-infra08:25
*** priteau has joined #openstack-infra08:29
*** ralonsoh has joined #openstack-infra08:30
*** ykarel|lunch is now known as ykarel08:38
*** hashar has joined #openstack-infra08:40
openstackgerritMerged openstack/project-config master: IRC #openstack-ironic gerritbot CI failed messages  https://review.opendev.org/69809108:43
*** rpittau|afk is now known as rpittau08:48
*** jpena|off is now known as jpena08:52
*** yamamoto has quit IRC08:53
*** dtantsur|afk is now known as dtantsur08:57
openstackgerritSimon Westphahl proposed zuul/nodepool master: Annotate logs in launcher  https://review.opendev.org/70355808:57
openstackgerritSimon Westphahl proposed zuul/nodepool master: Annotate logs in node request handler  https://review.opendev.org/70355908:57
openstackgerritSimon Westphahl proposed zuul/nodepool master: Include event id in node request listings  https://review.opendev.org/70356008:57
*** pkopec has joined #openstack-infra08:59
*** iurygregory has quit IRC08:59
*** florianf has quit IRC09:02
*** tosky has joined #openstack-infra09:07
openstackgerritSimon Westphahl proposed zuul/nodepool master: Annotate logs in zk module  https://review.opendev.org/70356109:10
*** goldyfruit has quit IRC09:11
*** tommylikehu has joined #openstack-infra09:11
openstackgerritJan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler  https://review.opendev.org/62147909:11
*** iurygregory has joined #openstack-infra09:13
*** yamamoto has joined #openstack-infra09:14
*** lucasagomes has joined #openstack-infra09:14
openstackgerritJan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler  https://review.opendev.org/62147909:15
*** yamamoto has quit IRC09:18
*** apetrich has joined #openstack-infra09:18
*** jaosorior has joined #openstack-infra09:19
*** gfidente has joined #openstack-infra09:23
*** xek has joined #openstack-infra09:24
*** xinranwang has quit IRC09:26
*** yolanda has quit IRC09:27
*** udesale_ has quit IRC09:28
*** SurajPatil has quit IRC09:28
*** udesale_ has joined #openstack-infra09:28
*** derekh has joined #openstack-infra09:28
*** SurajPatil has joined #openstack-infra09:28
*** SurajPatil has quit IRC09:29
*** surpatil has quit IRC09:29
*** surpatil has joined #openstack-infra09:30
*** yolanda has joined #openstack-infra09:33
openstackgerritAntoine Musso proposed zuul/zuul master: Docs: fix stestr run example  https://review.opendev.org/70356609:40
*** yolanda has quit IRC09:49
*** jaosorior has quit IRC09:54
*** tetsuro has quit IRC09:56
openstackgerritAntoine Musso proposed zuul/zuul master: tox: pass --slowest to stestr  https://review.opendev.org/70357109:58
openstackgerritAntoine Musso proposed zuul/zuul master: Divide concurrent tests by classes  https://review.opendev.org/70357510:08
*** ykarel is now known as ykarel|afk10:11
*** openstackgerrit has quit IRC10:12
*** priteau has quit IRC10:41
*** ykarel|afk is now known as ykarel10:43
*** goldyfruit has joined #openstack-infra10:47
*** udesale_ has quit IRC10:51
*** openstackgerrit has joined #openstack-infra11:10
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder master: Add ironic jobs to the CI  https://review.opendev.org/70247411:10
*** goldyfruit has quit IRC11:10
*** rpittau is now known as rpittau|bbl11:15
*** surpatil has quit IRC11:26
*** Wasaac has quit IRC11:32
*** Wasaac has joined #openstack-infra11:34
*** Lucas_Gray has joined #openstack-infra11:40
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder master: Add ironic jobs to the CI  https://review.opendev.org/70247411:49
*** rfolco has joined #openstack-infra12:00
*** dtantsur is now known as dtantsur|bbl12:01
AJaegerstevebaker, infra-root, https://review.opendev.org/#/c/698091/ merged to update IRC notifications for ironic but I don't see the new notifations at http://eavesdrop.openstack.org/irclogs/%23openstack-ironic/latest.log.html . Do we need to restart gerritbot? Or the change not working?12:06
*** priteau has joined #openstack-infra12:06
*** aedc has quit IRC12:09
*** aedc has joined #openstack-infra12:09
*** yolanda has joined #openstack-infra12:10
*** yolanda has quit IRC12:10
*** yolanda has joined #openstack-infra12:11
*** Wasaac has quit IRC12:12
*** ociuhandu has joined #openstack-infra12:12
*** Wasaac has joined #openstack-infra12:12
*** ociuhandu has quit IRC12:13
*** Lucas_Gray has quit IRC12:13
*** tkajinam has joined #openstack-infra12:14
*** Lucas_Gray has joined #openstack-infra12:17
*** TomStappaerts has joined #openstack-infra12:18
*** aedc has quit IRC12:20
*** aedc has joined #openstack-infra12:20
*** jpena is now known as jpena|lunch12:21
*** artom has joined #openstack-infra12:24
TomStappaertsHi guys, seems like pip 20 is breaking some (if not all) of our CI jobs?12:26
*** TomStappaerts has quit IRC12:27
*** TomStappaerts has joined #openstack-infra12:27
*** rcernin has quit IRC12:28
TomStappaertseg: https://81b633e2c5fe858f8400-d324a81a71d524d51ede3dc5aee27774.ssl.cf5.rackcdn.com/702831/4/check/networking-ovn-tempest-dsvm-ovs-release/c062094/12:29
tkajinamTomStappaerts, this one ? https://github.com/pypa/pip/issues/721712:29
*** TomStappaerts has quit IRC12:35
*** TomStappaerts has joined #openstack-infra12:36
*** dpawlik has joined #openstack-infra12:46
*** ykarel is now known as ykarel|afk12:47
*** lmiccini has quit IRC12:48
*** nicolasbock has joined #openstack-infra12:50
*** ociuhandu has joined #openstack-infra12:52
fricklerAJaeger: was there a -2 event since it merged? also very likely gerritbot needs restarting, I can look into that in a bit12:54
*** lseki has joined #openstack-infra12:57
*** udesale has joined #openstack-infra12:59
iurygregoryHey infra team, anyone aware of problems in stable branches such as "ImportError: cannot import name 'SourceDistribution'" ?13:01
*** ociuhandu has quit IRC13:01
tkajinamiurygregory, I'm not so familiar with the infra stuffs, but just sent an e-mail to share that error on ml13:01
yoctozeptoalso TomStappaerts has seen this issue13:02
tkajinamiurygregory, so hopefully somebody will see it and set pin on pip to fix the issue... I hope13:02
iurygregorytkajinam, tks, I'm trying to test locally to see if it's a problem on infra or not since we are getting a lot of FAILURE and POST_FAILURE in ironic CI13:03
yoctozeptothey fixed 19 mintues ago13:03
iurygregoryyoctozepto, tks!13:03
iurygregoryso a recheck would work?13:03
yoctozeptolooks like they b0rked something and quickly fixed13:03
yoctozeptoiurygregory: if it's the same issue they just fixed13:03
iurygregoryyoctozepto, ack I will try to trigger a recheck13:04
yoctozeptolooks like it, iurygregory, tkajinam, TomStappaerts13:04
*** priteau has quit IRC13:05
*** rpittau|bbl is now known as rpittau13:05
tkajinamyoctozepto ahhh, thanks.13:06
yoctozeptotkajinam: btw, which ml did you mean? nothing appearing on os-discuss...13:06
tkajinamI meant openstack-discuss13:06
*** lmiccini has joined #openstack-infra13:06
tkajinamyoctozepto, ^^^13:07
*** rlandy has joined #openstack-infra13:07
yoctozeptotkajinam: yeah, seeing now, thanks for confirming, must have been greylisted13:07
iurygregorythere is an email on openstack-discuss just now =)13:08
tkajinamiurygregory, yoctozepto good to heat that. I've not sent e-mail on that list for a long time, so I was a little bit afraid I made something wrong :-P13:08
tkajinamand I know see that follow-up mail was sent telling that the they fixed the issue in pip13:10
yoctozeptoI replied to close the topic13:10
tkajinamyoctozepto, thanks !!13:10
yoctozeptogood old pip likes to break from time to time :-)13:10
yoctozeptobut they have good response times, really13:11
tkajinamyeah it's surprisingly quick13:11
tkajinamthey were so quick that they didn't update the issue info before releasing the fix :-)13:12
yoctozeptotkajinam: indeed! well, if you break half the internets you first fix the issue, then post about it :-)13:12
tkajinamyoctozepto, yeah, that is much appreciated behavior.13:13
tkajinamall we have to do is to check commit logs first :-)13:13
tkajinamby the way I found an interesting file in devstack while looking for the way to pin pip13:14
yoctozeptoour natural habitat, wouldn't you say?13:14
tkajinamyoctozepto, definitely13:14
fricklero.k., I just watched one job pass the location where pip 20.0.0 broke things, so looks like the 20.0.1 fix is working13:14
yoctozeptodevstack is fun13:14
tkajinamfrickler, good to hear that news13:15
fricklerinfra-root: broken stream for http://zuul.openstack.org/stream/11236c64bee34787a854c896a005642f?logfile=console.log , maybe someone has time to check the executors later13:15
*** jamesmcarthur has joined #openstack-infra13:18
fricklerAJaeger: gerritbot was restarted at 10:12, well after the change merged13:18
*** zbr|drover has quit IRC13:18
*** zbr has joined #openstack-infra13:19
*** jpena|lunch is now known as jpena13:23
AJaegerfrickler: thanks for confirming13:24
AJaegerfrickler: I was looking for commented13:24
fricklerAJaeger: I think the patch is wrong, the event type is "comment-added", not "comments-added". I also don't find any "x-vrif-*" events13:27
*** whoami-rajat_ has joined #openstack-infra13:28
frickleroh, that's in https://review.opendev.org/#/c/698089/2/gerritbot/bot.py , will need to crosscheck the logs again13:29
frickleriiuc it needs to have the 'comment-added' tag in order to trigger that code13:30
fricklerI'm doing a follow-up patch13:31
*** exsdev0 has joined #openstack-infra13:32
*** AJaeger has quit IRC13:33
*** jamesmcarthur has quit IRC13:33
*** exsdev has quit IRC13:33
*** exsdev0 is now known as exsdev13:33
*** dpawlik has quit IRC13:33
*** icey has quit IRC13:33
*** AJaeger has joined #openstack-infra13:34
openstackgerritJens Harbott (frickler) proposed openstack/project-config master: Fix use of 'comment-added' event type  https://review.opendev.org/70361413:34
*** icey has joined #openstack-infra13:34
fricklerconfig-core: ^^13:34
*** iurygregory has quit IRC13:35
AJaegerthanks, frickler !13:38
*** Lucas_Gray has quit IRC13:42
sshnaidmis SourceDistribution error known?13:44
*** jamesmcarthur has joined #openstack-infra13:45
fricklersshnaidm: yes, pip 20.0.0 error. pip 20.0.1 was just released and fixes it13:45
sshnaidmfrickler, thanks!13:45
sshnaidmzbr, ^^13:45
sshnaidmweshay|ruck,  ^^13:46
*** ociuhandu has joined #openstack-infra13:47
*** ociuhandu has quit IRC13:53
*** aaronsheffield has joined #openstack-infra13:54
*** dtantsur|bbl is now known as dtantsur14:01
*** iurygregory has joined #openstack-infra14:05
fungiyoctozepto: tkajinam: we don't do greylisting on mailman, but posts to openstack-discuss take a few minutes to show up in folks inboxes just because of how many copies it needs to send out for its ~1300 subscribers14:15
tkajinamfungi, it makes sense. yeah I know we have many subscribes to the list.14:16
AJaegerfungi, could you review https://review.opendev.org/#/c/703614/, please? Quick IRC fix...14:17
tkajinamfungi, thanks for the info14:17
openstackgerritIlya Etingof proposed opendev/glean master: Fix a handful of bugs in config-drive processing  https://review.opendev.org/70362314:21
*** yamamoto has joined #openstack-infra14:21
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed  https://review.opendev.org/70362414:25
openstackgerritMerged openstack/project-config master: Fix use of 'comment-added' event type  https://review.opendev.org/70361414:29
*** yamamoto has quit IRC14:36
*** dtroyer has joined #openstack-infra14:38
*** kjackal has joined #openstack-infra14:46
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed  https://review.opendev.org/70363114:46
zbrclarkb: we are going to disable gzipping of logs on https://review.opendev.org/#/c/702862/ -- are you sure that this will not impact the log-servers?14:46
*** TomStappaerts has quit IRC14:56
*** TomStappaerts has joined #openstack-infra14:57
*** yamamoto has joined #openstack-infra14:59
*** yamamoto has quit IRC14:59
fungizbr: they are compressed on upload to swift14:59
*** yamamoto has joined #openstack-infra14:59
zbrfungi: mainly server side compression is tansparent, so we should not care.15:00
fungizbr: the only place i can think they might pose a problem is if they exceed the available transfer space on the executor during log collection15:00
fungiwhich is something i keep wondering about with this plan15:00
zbrfungi: we will find out, we can implement some truncation if needed.15:01
fungithen again, if there are super large logfiles, precompressing them may make sense because you're not going to view them with a browser anyway (if you're sane)15:01
zbrfungi: i agree. once we sport the first issue, i will try to implement extra logic.15:02
zbruntil then, we should be ok, I am not aware of normal huge files.15:02
*** yamamoto has quit IRC15:04
openstackgerritAntoine Musso proposed zuul/zuul master: tox: reduce deps used for pep8 env  https://review.opendev.org/70363415:05
*** kjackal has quit IRC15:06
openstackgerritAntoine Musso proposed zuul/zuul master: tox: rename pep8 to linters  https://review.opendev.org/70363515:13
openstackgerritAntoine Musso proposed zuul/zuul master: tox: do not install bindep for linters  https://review.opendev.org/70363615:13
*** tkajinam has quit IRC15:16
*** electrofelix has joined #openstack-infra15:16
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed  https://review.opendev.org/70362415:17
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add networking.k8s.io apiGroups rbac for service account  https://review.opendev.org/70363715:17
*** jamesmcarthur has quit IRC15:25
*** kjackal has joined #openstack-infra15:30
*** ociuhandu has joined #openstack-infra15:30
*** electrofelix has quit IRC15:31
noonedeadpunkhey everyone:)15:32
noonedeadpunkhow can I ask for a job hold to look into what's happening in vm? For instance for that build? https://zuul.opendev.org/t/openstack/build/021f80901599483ab2e28c977506b67715:33
*** jamesmcarthur has joined #openstack-infra15:35
*** electrofelix has joined #openstack-infra15:36
*** electrofelix has quit IRC15:36
*** electrofelix has joined #openstack-infra15:36
openstackgerritTobias Henkel proposed zuul/zuul master: Add spec for enhanced regional executor distribution  https://review.opendev.org/66341315:38
funginoonedeadpunk: looks like that job was successful... what's the issue you're investigating?15:39
openstackgerritTobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs  https://review.opendev.org/67384015:39
AJaegernoonedeadpunk: we can only hold a job that fails...15:39
noonedeadpunkAh, I see.15:39
noonedeadpunkLike it's just missing gathered data, so I was thinking I would be able to get it that way...15:40
fungiyep, i just checked the output of `zuul autohold --help` and we don't seem to have an option for holding on success or on non-failure results15:40
noonedeadpunkactually then https://zuul.opendev.org/t/openstack/build/76dc975e3ea74ac3a2a8ff1d791f1633 would be helpful as well15:40
noonedeadpunkit what I'm actually investigatig :p15:41
*** Lucas_Gray has joined #openstack-infra15:42
fungiremember that you can of course usually extend the job to run whatever commands you're hoping to run manually to poke around on the filesystem15:42
jrosserfollowing up the neutron-lib adventure from yesterday, will the existing wrong .py2.py3 wheels persist in the wheel cache?15:44
funginoonedeadpunk: zuul autohold --tenant openstack --project openstack/openstack-ansible-os_manila --job openstack-ansible-deploy-aio_metal-ubuntu-bionic --change 675934 --reason "noonedeadpunk investigating btrfs-related issues for lxc" --count 115:45
fungidoes that capture what you're looking for?15:45
*** lpetrut has quit IRC15:46
fungijrosser: if https://review.opendev.org/703487 works (i haven't checked a recent build log yet but will shortly) then we can safely delete that wheel from our cache and it shouldn't reappear15:46
funginoonedeadpunk: i've set that autohold if you want to recheck change 67593415:47
funginoonedeadpunk: if i should make adjustments to the autohold to better capture what you're looking for, let me know15:48
*** chandankumar is now known as raukadah15:48
noonedeadpunkOh, I've started searching for the way I should launch that :p15:48
clarkbfungi: re precompressing making sense for large files, that is why we xz the serialized journal file. its like half a gig uncompressed and you have to pass it through journald anyway to view it so may as well compress it down to like 30MB15:48
funginoonedeadpunk: once that job fails again i can ssh into the node and add your public ssh key for root access15:49
noonedeadpunkfungi: oh, thanks! It's just the first time I'm asking for the hold, so wasn't sure how it works:)15:49
noonedeadpunkthanks for explaining15:50
*** ociuhandu has quit IRC15:51
AJaegerconfig-core, infra-root, I created a change for system-config to not publish doctrees ( https://review.opendev.org/703544 ) but then thought, let's fix it for every job with https://review.opendev.org/703547 in zuul-jobs. What do you think?15:52
clarkbAJaeger: ++ to fixing globally15:53
corvuswhat's a .doctree dir?15:54
*** zxiiro has joined #openstack-infra15:57
*** jtomasek has quit IRC15:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test  https://review.opendev.org/70275815:58
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed  https://review.opendev.org/70362415:58
*** udesale has quit IRC15:59
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Enable E741 flake8 check  https://review.opendev.org/70365016:00
corvusShrews: can we just make that change without enabling that check?16:01
corvusi think we can make a judgement call that "for x in list" is fine.16:01
*** iurygregory has quit IRC16:01
*** hashar has quit IRC16:02
*** auristor has joined #openstack-infra16:04
fungitechnically e741 doesn't mind variables named "x"16:08
corvusoh, is this the "my font doesn't show the difference between l and 1" check?16:08
fungiit specifically cares that variables are not named l, I or O due to typographical similarities with digits 1 and 016:09
fungii agree the check is of questionable value, but its reach is fairly limited16:09
corvussorry i misremembered the check16:09
Shrewsit's not an important change. we can live without it16:12
openstackgerritMerged zuul/zuul-jobs master: fetch-sphinx: Exclude doctrees directory  https://review.opendev.org/70354716:12
*** jackedin has joined #openstack-infra16:13
*** openstackgerrit has quit IRC16:13
*** openstackgerrit has joined #openstack-infra16:14
openstackgerritClément Mondion proposed zuul/nodepool master: add tags support for aws provider  https://review.opendev.org/70365116:14
jrosserive just had two jobs fail with unable to get to opendev git repos "fatal: unable to access 'https://opendev.org/openstack/cinder/': Encountered end of file" and fatal: unable to access 'https://opendev.org/openstack/neutron/': GnuTLS recv error (-110): The TLS connection was non-properly terminated."16:18
jrosserboth on here https://review.opendev.org/70338916:19
clarkbjrosser: it helps if you can link to the job logs16:19
*** ykarel|afk is now known as ykarel16:20
clarkbas a first pass sanity check the http top level page for neutron and cinder loads for me from all 8 backends16:22
jrosserok, normally i'd just recheck but two together like that looks suspicious16:22
clarkbjrosser: out of curiousity why are those jobs cloning the repos and not using the zuul supplied repos?16:22
*** ociuhandu has joined #openstack-infra16:23
clarkbgoing to try git clones against all backends now16:23
*** jtomasek has joined #openstack-infra16:26
jrosserclarkb: i don't have a better answer than "it'd be complicated", it's been like this forever so there must be some justification16:26
clarkbok, we go out of our way to ensure that our CI jobs don't become a self inflicted DDoS16:27
jrosserbecause the repos of osa itself are definately picked up from the zuul supplied ones16:27
clarkbjobs should ideally only clone direclty if they are testing that those clones work16:27
*** ociuhandu has quit IRC16:28
clarkbI'm still digging around to see if there is a smoking gun for this, but we recognize this as a faulty configuration and have alternatives in place as a result16:28
*** lmiccini has quit IRC16:30
clarkbgitea04 appears to have been hit erally hard during that period and ran out of memory, swapped, and had high load average as a result16:30
noonedeadpunkfungi: it has failed, but not where I was expecting...16:31
clarkbI expect that is the cause of the problem16:31
noonedeadpunkbut it might be enough actually16:31
jrosserclarkb: i will try to find out why it is like this16:32
funginoonedeadpunk: okay, where can i get a copy of your public ssh key?16:32
*** tosky has quit IRC16:33
funginoonedeadpunk: ssh root@
noonedeadpunkfungi: thanks!16:34
*** eharney has quit IRC16:35
openstackgerritClément Mondion proposed zuul/nodepool master: add tags support for aws provider  https://review.opendev.org/70365116:35
fungiclarkb: oh, yeah, check out the swap graph16:35
fungi[Tue Jan 21 15:00:50 2020] Out of memory: Kill process 11821 (gitea) score 817 or sacrifice child16:36
*** ociuhandu has joined #openstack-infra16:36
fungii guess the health checks we added to haproxy aren't sophisticated enough to catch that16:37
clarkbfungi: ya I'm looking at haproxy logs for connections to gitea04 between 1300UTC and 1600UTC to see if I can make sense of what may have caused it16:37
clarkbfungi: I think they are, but any existing connections would be lost16:37
fungiahh, okay, so maybe impact was somewhat limited16:37
*** gyee has joined #openstack-infra16:39
zbris by design or bug that .sh files are downloaded while .bash ones are loaded as text? (logs)16:40
*** iurygregory has joined #openstack-infra16:40
clarkbzbr: we use a python mimetype lib to figure that out iirc16:41
clarkbzbr: possibly a bug in that tool16:41
*** mattw4 has joined #openstack-infra16:41
clarkbwhere it decides .bash is a text type and .sh isn't?16:41
zbrclarkb: thanks, i will look into, i guess everyone wants to be able to see them without downloading them.16:42
weshay|ruckzbr, k.. so we're fixing here? cool16:43
*** jpena is now known as jpena|brb16:46
*** bnemec has quit IRC16:48
*** tommylikehu has quit IRC16:48
fungiinfra-root: if anyone has a moment to approve a new ml request from last week for lists.opendev.org, these folks are hoping to start using it soon to plan an upcoming collaboration at the mass open cloud workshop: https://review.opendev.org/70314516:51
clarkbfungi: I've +2'd but not approved in case we can get another non staffer to review, but probably fine to proceed16:52
corvusi wondere where we are on our MOC credentials16:56
clarkbthere are a lot of huawei IPs hitting gitea0416:56
* clarkb tries to finish cleaning up this data so it is easier to understand16:57
*** kjackal has quit IRC16:58
fungimordred: if you're around, have you heard any more about moc creds for nodepool?16:59
fungiyou're probably asleep right now though17:00
*** tesseract has quit IRC17:01
*** rpittau is now known as rpittau|afk17:04
*** ricolin has quit IRC17:04
*** pgaxatte has quit IRC17:04
*** lucasagomes has quit IRC17:05
*** ricolin has joined #openstack-infra17:05
clarkbhuawai was almost half the total connections during that ~hour period where gitea04 was swapping17:10
clarkbred hat nat has most connections for a single IP17:11
clarkbI expect that the huawei IPs are a CI system (maybe openlab?) they are all within like the same /2217:12
clarkbanyone know who to talk to at openlab now? smcginnis maybe?17:12
clarkbit would be great if other CI systems didn't DDoS us in addition to our efforts to rpevent our CI system from DDoSing us17:12
clarkbjrosser: ^ fyi that is my quick read of the haproxy logs17:12
jrosserclarkb: we just had a loooong discussion about this in #openstack-ansible17:13
jrosserand its a sort of mashup of "history" and "it needs to behave like it would for end users"17:14
*** jtomasek has quit IRC17:14
clarkbjrosser: the way we address "behave like it would for end users" is to have a pre step put the git repos in place, then the job/tool/whatever only clones if that repo isn't alredy there17:14
clarkbjrosser: this should work for CI and not CI17:14
jrosserbut thats not to say we can't look at moving to the zuul cloned repos, but i'm a bit wary of never testing the code path that folk in the wild would use17:14
clarkbjrosser: also that prevents you from doing cross project testing17:15
clarkbwhich is a very powerful tool17:15
fungithe concern is that you would cease testing whether git breaks its ability to do a git clone?17:15
fungii hope the git maintainers test their code17:15
fungiand don't rely on us to test that for them17:16
*** rfolco is now known as rfolco|brb17:16
clarkbother advantages include being able to run mutliple times without incurring clone costs (or failing beacuse code is already there). Also users can preset up their git repos this way as well if they know they want a specific version of something17:16
fungiyeah, they gain the ability to set up wiregapped environments that way17:17
clarkbI need to find breakfast but as far as addressing the OOM I think we either A) make bigger gitea nodes and/or B) ask huawei (maybe it is openlab) to use cached repos17:17
fungier, airgapped i mean17:17
fungiyeah, i guess since there's no separate webserver in front of gitea on the same host, we don't have the ability to perform resource management throttles17:18
clarkboh there is a C) try the least conns lb method again17:18
fungii'm wary of least connections until we have shared backend clustering working17:19
clarkbya it will probably make some clients unhappy again if the pack files get out of sync17:19
clarkb(I think pack vs object was causing the problems before because if you think the file is in a pack or an object then request the other you'll fail?)17:20
fungieven if packfiles don't get out of sync, clients are racing replication events from gerrit17:20
openstackgerritMerged opendev/system-config master: Add mailing list for OpenInfra Labs  https://review.opendev.org/70314517:20
fungiinvariably, some fetches will go to backends which don't have those refs yet17:20
clarkbthats a good point17:21
fungigerrit doesn't provide any guarantees that the same refs are replicated to the same destinations at the exact same times17:22
smcginnisclarkb: That's probably Huawei proper and not OpenLab. OpenLab (at least was) spread out across different providers.17:22
smcginnisNot sure who would be a good contact there now.17:23
smcginnisMaybe mnaser is still working with them on some things?17:23
smcginnisMost of the team was still all Huawei employees spread out between India and China.17:23
openstackgerritClément Mondion proposed zuul/nodepool master: add tags support for aws provider  https://review.opendev.org/70365117:23
fungiclarkb: what about request rate limiting or bandwidth throttles per ip address in haproxy?17:24
fungiwe might be able to tune it so that only addresses which are overusing the git farm get poor performance17:25
*** jpena|brb is now known as jpena17:26
AJaegerianw: I think you have some packaging background as well, could you review https://review.opendev.org/#/c/703495, please?17:26
fungithough i know bandwidth utilization and request frequency don't directly map to memory utilization on the backend (that depends a lot on the request type)17:26
clarkbfungi: that might work butmay make it worse since the impact is memory by git operations. Completing those as quickly as possible is best17:26
*** jtomasek has joined #openstack-infra17:26
clarkbslowing them down will only increase memory demand I think17:26
fungithat's a great point17:26
fungisounds more like a feature request for gitea itself: client rate limits or some sort of resource management so it can start rejecting connections when it's overloaded17:27
*** gfidente is now known as gfidente|dinner17:29
fungiAJaeger: specifically, anyone with background on how setuptools/pip shells out to gcc for compiling python extensions would be a huge help. i wasn't having a lot of luck tracking down where/how that happens17:31
fungiianw: ^17:31
clarkbsmcginnis: thanks for the info. Maybe we can reach out via our board member there if this persists17:32
AJaegerfungi: yeah, couldn't find it either.17:32
clarkbI think its up to the package?17:34
*** evrardjp has quit IRC17:34
clarkbit has been a long time since I fiddled with C linked packages though17:34
*** evrardjp has joined #openstack-infra17:34
fungiyeah, i was looking at pyyaml as my initial example and couldn't track down how "pip install pyyaml" or more specifically "pip wheel pyyaml" (which should be the same codepath for that part) winds up compiling the libyaml extension .so file17:36
openstackgerritAntoine Musso proposed zuul/zuul master: tox: do not install bindep for linters  https://review.opendev.org/70363617:36
openstackgerritMerged zuul/zuul master: Docs: fix stestr run example  https://review.opendev.org/70356617:41
AJaegerconfig-core, do we want all openstack-tox-p36/37 jobs to increase timeout to 1h, or ask neutron team to do that in-repo? See https://review.opendev.org/70338617:42
AJaegerit's 40 minutes now - and they run sometimes in timeouts17:43
clarkbI think I'm ok with a global setting but I havent checked runtimes of jobs to seehow close others are to that limit. I expect nova is close too17:44
*** TomStappaerts has quit IRC17:44
*** eharney has joined #openstack-infra17:46
AJaegernova has ~15 mins17:46
*** ykarel is now known as ykarel|away17:47
clarkboh wow that is quicker than I though17:48
AJaegeryep - looking at http://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py37 now...17:49
AJaegercyborg timed out17:49
AJaegercyborg, neutron, keystonemiddleware is what I see17:50
*** iurygregory has quit IRC17:51
*** harlowja has quit IRC17:51
*** harlowja has joined #openstack-infra17:52
*** yolanda has quit IRC17:52
*** yolanda has joined #openstack-infra17:54
fungii wonder if nova's were sped up by the mox to mock transition17:59
fungithey used to run a lot longer17:59
*** jamesmcarthur has quit IRC18:00
*** roman_g has quit IRC18:01
*** TomStappaerts has joined #openstack-infra18:02
fungiWOAH, i wasn't getting a response from lists.o.o for longer than seemed healthy. i finally managed to ssh in and its load average is over 10018:08
fungithere may be another subscription spamming event underway18:08
fungidigging into it now18:08
clarkbfungi: thank you. Let me know if I can help18:09
clarkbotherwise I'm going to catch up on the governance change reviews now18:09
fungithough there were tons of www-data owned python processes so it might be mass crawling of pipermail archives on it18:09
*** dtantsur is now known as dtantsur|afk18:11
*** jtomasek has quit IRC18:13
*** ociuhandu_ has joined #openstack-infra18:13
*** TomStappaerts has quit IRC18:14
*** Lucas_Gray has quit IRC18:15
openstackgerritClark Boylan proposed opendev/system-config master: Update project doc to reflect OpenDev changes  https://review.opendev.org/70348818:15
clarkbfrickler: ^ thank you for the reviews18:15
*** ociuhandu has quit IRC18:17
fungino sign of mass subscription activity18:17
*** ociuhandu_ has quit IRC18:18
fungithere was a spike in web requests to the openstack.org vhost but that started after the load average cleared up18:18
fungialso i think our ansible wheel may be getting bogged down by something18:19
fungithe last entry in /var/log/ansible/run_all_cron.log was 17:01:45z18:19
fungiso over an hour ago18:20
fungilooks like it's been trying to ssh to the ipv6 address corresponding to zm06.openstack.org for all that time18:21
fungii'll see what's up (or down rather) with zm0618:21
clarkbianw: small thing on https://review.opendev.org/#/c/703535/3 otherwise that looks good to go18:21
fungizm06 responds to icmp ping but not ssh18:22
fungimight be something nasty has happened to its rootfs. will try to check out the oob console for any i/o errors from the kerneol18:22
fungiINFO: task jbd2/xvda1-8:309 blocked for more than 120 seconds.18:25
fungilooks like the rootfs to me18:26
fungii'll try to force reboot it18:26
*** slaweq_ has joined #openstack-infra18:26
*** slaweq has quit IRC18:27
fungilooks like the last time cacti was able to get a response from zm06 was 2020-01-16:05:00z18:28
fungiso ~5.5 days ago18:28
clarkbI think that is what we see from live migrations18:28
fungi#status log performed a hard reboot of zm06 after it lost the use of its rootfs (likely 2020-01-16:05:00z per gap in cacti graphs)18:29
openstackstatusfungi: finished logging18:29
fungiit's up now18:29
*** ralonsoh has quit IRC18:31
fungizuul-merger service won't start either... lockfile.LockFailed: failed to create /var/run/zuul/merger.pid18:32
clarkbprobably leaked that file18:32
fungithere's a /var/run/zuul-merger directory but no /var/run/zuul directory18:32
fungidid we change initscripts recently?18:33
fungior puppetry?18:33
fungilooks like zm05 has both18:33
clarkbnot that I was aware of. I seem to recall /var/run/zuul/merger.pid being the correct location. We also restarted zuul semi recnetly and owuld've expected that to fail18:33
*** electrofelix has quit IRC18:34
clarkb/var/run is not persistent fs though iirc18:34
clarkbpossible you need a puppet pulse to create that dir then it can start the service18:34
*** jpena is now known as jpena|off18:34
fungion zm05 /var/run/zuul-merger was modified 2019-01-09 and /var/run/zuul was last modified 2019-01-1418:34
fungiso we must have switched which directory we're using between a restart on the 9th and a restart on the 14th?18:35
clarkbsemi related, zuul schedulers memory has been stable since we reverted that change18:35
clarkbcorvus: ^ we should probably consider reverting it soon if this holds up?18:35
clarkbfungi: no this change was a long time ago iirc18:35
clarkbrun/zuul-merger was the old location then we consolidated everything in run/zuul/18:36
clarkbbut run/zuul is wiped on every reboot so we have to wait for puppet to run to write it down iirc18:36
fungiclarkb: well, what's odd is that zm06 has a /var/run/zuul-merger created/modified at the time i tried to start the zuul-merger service but then it tries to lock a pidfile in /var/run/zuul which dne18:36
clarkbwe might want to have the init script write the dir if not there as an alternative18:36
clarkbfungi: ya I think the init script wasn't updated to create the proper dirwhen they changed a while back18:37
clarkbpuppet was though18:37
clarkbya we need to update the PIDFILE arg of the init script18:38
clarkblooks like the other services need similar help18:39
*** ramishra has quit IRC18:41
corvusclarkb: ack, i'll propose the revert, thx18:50
fungii'll work on the pidfile handling fix for the initscript now18:51
openstackgerritJames E. Blair proposed zuul/zuul master: Revert "Extract an abstract base Parser class"  https://review.opendev.org/70366918:52
fungiinfra-root: i manually killed the hung ssh process on bridge.o.o which was trying to reach the (since rebooted) zm06, and ansible has continued past it now18:55
fricklerFYI if someone comes along with weird pip failures in devstack master, this got merged half an hour ago and might be the cause openstack/devstack master: Revert "Do not use pip 10 or higher"  https://review.opendev.org/56159719:09
clarkbfrickler: thank you for the heads up19:09
fungithat's been a long time coming19:12
*** dustinc|PTO is now known as dustinc19:13
*** jamesmcarthur has joined #openstack-infra19:20
stevebakerfrickler, AJaeger: thanks for the comment-added followup19:25
*** tosky has joined #openstack-infra19:25
*** jackedin has quit IRC19:27
*** iurygregory has joined #openstack-infra19:33
openstackgerritMerged opendev/storyboard-webclient master: Remove unused imagemin build step  https://review.opendev.org/69105019:34
iurygregoryHello Infra o/  is there any know issues with openstack-tox-* jobs? in ironic I noticed a lot of FAILURE / RETRY_LIMIT for this jobs in in master and stable branches19:35
clarkbiurygregory: I am not aware of any known issues. Examples (links to logs) can be helpful19:37
fungiiurygregory: were they from much earlier today? there was a broken pip release for a few hours19:37
clarkboh huh I missed that but that would cause retry limits if pip fails early in the job19:37
fungipip 20.0.0 is bad, 20.0.1 solved it19:37
frickleriurygregory: a couple of hours there was an issue caused by pip 20.0.0, which should now be fixed by 20.0.1, some error with SourceDistribution?19:37
iurygregoryhttps://review.opendev.org/703381 https://review.opendev.org/70338019:37
openstackgerritMerged opendev/storyboard-webclient master: Update selenium-standalone and gifsicle  https://review.opendev.org/69105119:37
openstackgerritMerged opendev/storyboard-webclient master: Reinstate "Add transpiling as a step in the build process"  https://review.opendev.org/69147719:37
iurygregoryI got this one earlier but now seems to bee different19:37
fungiiurygregory: some of us can probably dig deeper on that when the infra meeting finishes19:38
iurygregoryfungi, tks, i will try to keep the irc open in my notebook o/19:39
fricklerlooks like some other pip issue https://zuul.opendev.org/t/openstack/build/5a6e757cd7024b5cba39f0bf64f77efe/console#2/0/0/ubuntu-bionic19:41
*** gfidente|dinner is now known as gfidente19:44
openstackgerritIan Wienand proposed opendev/system-config master: Add Linaro US cloud  https://review.opendev.org/70353519:44
ianwclarkb: ^ also found another typo ... i feel sure we could more programatically create some of these files19:45
ianwprobably falls into the cost-of-automating v new clouds actually added equation though19:45
clarkbianw: if the clouds run horizon I think it will output a clouds.yaml to you19:46
*** jamesmcarthur has quit IRC19:46
prometheanfirelooks like requirements bot is failing, still with the tempest issues?19:46
fungifrickler: that looks suspiciously like something which just changed in the zuul-jobs ensure-pip role19:48
fungifrickler: zbr: https://review.opendev.org/702978 merged 4 days ago which added that logic19:50
*** jamesmcarthur has joined #openstack-infra19:50
openstackgerritSorin Sbarnea proposed opendev/gear master: packaging: updated project urls  https://review.opendev.org/70342219:50
fungiensure-tox not ensure-pip i mean19:50
fungino stdout nor stderr but returns an exit code of 14119:51
zbrinteresting error19:53
zbrafaik that was command not found or something similar19:53
zbris related to pipefail https://stackoverflow.com/questions/22464786/ignoring-bash-pipefail-for-error-code-14119:54
*** smarcet has joined #openstack-infra19:55
*** nicolasbock has quit IRC19:57
zbris there any chance that /bin/bash not being real bash?19:58
ianwno, if it's explicitly called like that it's bash19:58
zbrtbh, is would be possible to avoid the pipefail if we want, it not really the most complex pieace of bash I seen.19:58
zbrwhat is interesting is that I am unable to replicate the failure manually19:59
fungiany idea if it's happening reproducibly or just intermittently?19:59
fungiyeah, i tried the same code locally in an interactive shell and wasn't seeing any issues with it19:59
openstackgerritDavid Shrewsbury proposed zuul/zuul-jobs master: ensure-tox: Output tox version  https://review.opendev.org/70123620:00
*** jamesmcarthur has quit IRC20:01
zbri would remove pipefail, add -x, just in case.20:01
*** jamesmcarthur has joined #openstack-infra20:01
zbrbut i confess, is frustrating to look at this task output and to guess what could have went wrong.20:03
zbrit seems like an one-off, based on 70338120:03
zbri think i remember, that happened when ssh connection dropped, mainly the error indicates that one of std??? stream was closed prematurely20:05
zbri am sure i seen the same error before20:05
ianwit doesn't seem like "command -v" could return anything other than 0/120:06
*** hashar has joined #openstack-infra20:07
*** jamesmcarthur has quit IRC20:08
*** jtomasek has joined #openstack-infra20:10
fricklerif both commands are installed, "command -v pip pip3" outputs two lines, but "head -n1" exists after the first is processed20:10
openstackgerritMerged zuul/zuul-website master: Remove some redirects  https://review.opendev.org/70345720:11
fricklerso there may be a race that sometimes causes command output to trigger a EPIPE20:11
*** yamamoto has joined #openstack-infra20:11
openstackgerritClark Boylan proposed zuul/zuul-website master: Fix releasenotes redirects  https://review.opendev.org/70368720:14
*** jamesmcarthur has joined #openstack-infra20:14
ianwfrickler: yeah ...20:15
ianw$ while [ 1 ] ; do bash ./set.sh; r=$?; if [[ $r != 0 ]]; then echo $r; fi; done20:15
*** yamamoto has quit IRC20:16
ianwthat's a very interesting trap, not sure i've seen that before20:16
ianwfrickler: are you writing a change?20:17
openstackgerritClark Boylan proposed zuul/zuul master: Speed up ansible plugin tests  https://review.opendev.org/70368820:19
fricklerianw: no I'm off for today, feel free to take over20:19
ianwfrickler: np, ttyl!20:23
*** Lucas_Gray has joined #openstack-infra20:26
*** eharney has quit IRC20:27
smarcetfungi: afternoon, could u stop the puppet agent for openstackid production ? i would need to update puppet script and would like to test first at dev :)20:29
*** yolanda has quit IRC20:29
openstackgerritClark Boylan proposed zuul/zuul master: Speed up ansible plugin tests  https://review.opendev.org/70368820:29
*** yolanda has joined #openstack-infra20:30
openstackgerritMerged zuul/zuul master: tox: pass --slowest to stestr  https://review.opendev.org/70357120:33
fungismarcet: i have added openstackid01.openstack.org (the server which hosts the openstackid.org site) to our emergency disable list for ansible so it will not run puppet there further until we remove it from the list again20:34
smarcetfungi: thx u :)20:34
fungiianw: frickler: zbr: maybe... echo `command -v pip pip3` | cut -d' ' -f120:36
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: fix pipe race  https://review.opendev.org/70368920:36
fungialso possible putting the first command in a subshell could work around it?20:36
ianwfungi: heh, or maybe ^ ?  two things i thought ... 1) is that we could have a "else echo "i could not find pip"; exit 1" ... but that seems possibly backwards incompatible because it would currently work without pip, *if* tox was installed anyway20:37
ianwand 2, should we use pip3 in preference to pip if both are found?20:37
ianwagain, backwards incompat change maybe, but it is 2020 ...20:37
fungivery good points20:37
openstackgerritMerged zuul/zuul master: doc: add links to components documentation  https://review.opendev.org/70310520:38
fungireversing the parameter order in the command command solves which one appears first, btw20:39
*** rcernin has joined #openstack-infra20:42
openstackgerritFelipe Reyes proposed openstack/project-config master: Add charm-interface-keystone-notifications project  https://review.opendev.org/70369120:43
openstackgerritFelipe Reyes proposed openstack/project-config master: Add charm-interface-keystone-notifications project  https://review.opendev.org/70369120:44
*** yolanda has quit IRC20:46
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: use pip3 in preference to pip  https://review.opendev.org/70369420:46
*** yolanda has joined #openstack-infra20:50
zbrfungi: backtick syntax is discouraged20:50
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: fix pipe race  https://review.opendev.org/70368920:50
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-tox: use pip3 in preference to pip  https://review.opendev.org/70369420:50
ianwfungi: if you get a sec for the arm64 cloud https://review.opendev.org/#/c/703535/ i'll watch it on bridge today, to be sure i didn't typo anything in the host_vars bit on bridge.o.o (someone can check bef2aeda30f496e4b32523dcc3e85d0b269dc505 on bridge)20:54
openstackgerritJames E. Blair proposed zuul/zuul-website master: Update redirects  https://review.opendev.org/70345620:56
*** whoami-rajat_ has quit IRC20:58
*** jamesmcarthur has quit IRC20:58
fungizbr: consider it shorthand for $(), point is to evaluate in a subshell so its output is not truncated while it's running21:02
openstackgerritMerged zuul/zuul master: Limit parallelity when installing ansible  https://review.opendev.org/70312621:04
*** harlowja has quit IRC21:04
openstackgerritMerged zuul/zuul-website master: Update redirects  https://review.opendev.org/70345621:07
*** kjackal has joined #openstack-infra21:08
openstackgerritAntoine Musso proposed x/gearman-plugin master: Add maven/java8 jdk as test bindeps  https://review.opendev.org/51828421:10
*** smarcet has quit IRC21:15
*** kjackal has quit IRC21:18
openstackgerritMerged opendev/gear master: packaging: updated project urls  https://review.opendev.org/70342221:19
openstackgerritMerged zuul/zuul-jobs master: ensure-tox: fix pipe race  https://review.opendev.org/70368921:20
*** armax has joined #openstack-infra21:22
*** diablo_rojo has quit IRC21:27
*** diablo_rojo has joined #openstack-infra21:28
*** eharney has joined #openstack-infra21:28
openstackgerritMerged zuul/zuul master: Docs: change "config" title  https://review.opendev.org/70347121:29
openstackgerritMerged zuul/zuul-jobs master: ensure-tox: Output tox version  https://review.opendev.org/70123621:32
*** jamesmcarthur has joined #openstack-infra21:35
*** gfidente has quit IRC21:36
*** jtomasek has quit IRC21:38
openstackgerritMerged zuul/zuul master: docs: improve job.role documentation  https://review.opendev.org/70337221:51
openstackgerritAntoine Musso proposed zuul/zuul master: test_repo_repr does not need to clone  https://review.opendev.org/70369822:02
fungiianw: if you get a moment, can you take a look at 703495? it's still not entirely clear to me what AJaeger is saying about setting CFLAGS in the script causing optimizations to not be applied when the compiler is invoked22:05
*** mattw4 has quit IRC22:06
*** eharney has quit IRC22:06
*** mattw4 has joined #openstack-infra22:06
ianwfungi: was just looking :)  i think the concern might be, and i've certainly seen it before, that you set CFLAGS=... and it completely overrides whatever defaults something like autoconf/make provide22:08
ianwwhich i should say i think is right, if you're trying to debug something and set CFLAGS= and it goes off and does it's own thing, that's very annoying22:13
openstackgerritAntoine Musso proposed zuul/zuul master: tests: remove test_repo_repr  https://review.opendev.org/70369822:14
*** mattw4 has quit IRC22:14
fungias in makefiles will only conditionally apply (some) compiler flags in the absence of a CFLAGS value?22:15
*** mattw4 has joined #openstack-infra22:15
fungirather than appending the supplied CFLAGS to the compiler flags they would normally apply?22:15
fungii'm going to guess this is highly project-dependent22:16
*** slaweq_ has quit IRC22:16
fungiand means that my decades-old assumptions about how CFLAGS is used in practice is now very dated22:16
*** pkopec has quit IRC22:18
fungiaha, looks like a number of projects "standardized" XCFLAGS (and XLIBS) as an alternative22:19
ianwyeah, i think an additional complication might be that some things try to replicate the build flags that the python library it's building against were built with22:20
fungi" Some package install scripts, like SDL, allow CFLAGS settings to override their normal settings (instead of append to them), so setting CFLAGS can cause harm in this case." https://en.wikipedia.org/wiki/CFLAGS22:23
* fungi grumbles22:23
ianwi really only have experience with automake, where IIRC you're supposed to use AM_ flags for "invariant" things, things that would "cause harm" if modified22:25
ianwthen things like optimize flags go in CFLAGS, so if the user wants them off they override22:26
fungii'm honestly not even sure whether c extensions for python modules regularly rely on autotools22:27
fungiwe observed in the pyyaml case at least that exporting CFLAGS in the environment did not result in a removal of unique gcc options (though without exporting CFLAGS there were a number of redundant options appearing in the command line)22:28
ianwno, i'm sure they don't -- i don't have much experience on that side22:28
*** hwoarang has quit IRC22:28
fungibut i had little luck tracking down where the cc was getting invoked22:29
fungiso many layers of tooling22:29
*** hwoarang has joined #openstack-infra22:30
fungiclarkb: corvus: after a while of looking at the puppet-zuul module, i think the problem may be that the zuulv3 initscript variants didn't get the logic from the v2 initscripts to create pidfile dirs. do you happen to know if there was a specific reason?22:39
fungier, rather, they do but they hard-code a specific path and didn't get https://review.openstack.org/53082022:40
fungii think that's the real problem22:40
corvusfungi: i don't recall that being intentional22:41
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add OpenShift SCC and functional test  https://review.opendev.org/70275822:41
fungiahh, yep, they seem to have raced that improvement22:41
fungithe v3 initscripts were created before that fix merged to the v2 initscripts22:41
fungiokay, i'll just port it so we start creating the correct pidfile directories22:42
*** jamesmcarthur has quit IRC22:42
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Handle service restart when connections are changed  https://review.opendev.org/70362422:46
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add tenant reconfiguration when main.yaml changed  https://review.opendev.org/70363122:48
clarkbfungi: note the pidfule var would still be wrong there I think22:50
clarkbsince $NAME is eg zuul-executor and what we want is /var/run/zuul/executor.pid iirc22:51
openstackgerritJeremy Stanley proposed opendev/puppet-zuul master: Make v3 pidfile directory creation dynamic  https://review.opendev.org/70370522:51
fungiclarkb: we override it with /etc/default/*22:51
fungiso the underlying issue seems to simply be that the hard-coded path for the directory to create isn't the parent directory of the pidfile value we override with in /etc/defaults22:52
fungianyway, there's the fix, i think, so hopefully it doesn't bite us on future reboots22:54
*** aaronsheffield has quit IRC22:54
fungiand you were right that puppet would eventually create the correct directory for us (we could probably delete that from the manifest now?) so i've started zuul-merger successfully on zm06 again22:55
*** tkajinam has joined #openstack-infra22:57
openstackgerritMerged zuul/zuul master: Revert "Extract an abstract base Parser class"  https://review.opendev.org/70366923:01
corvus=win 1223:05
clarkbI'll review that change just as soon as my laptop starts cooperating again23:07
*** Lucas_Gray has quit IRC23:08
*** Lucas_Gray has joined #openstack-infra23:09
fungii don't think there's any hurry23:09
fungiit's an infrequent annoyance at most, i just wanted to get to the bottom of it23:10
clarkbya but if we don't fix it now we'll forget until next time :)23:10
fungiwell, worst case it sits in review and gets ignored until we hit the problem again, forget there's a fix already written, repeat the investigation, push up a new fix, then see the review conflict ;)23:11
fungi(not the first time that's happened to me... this week)23:11
*** hashar has quit IRC23:16
ianwfungi: ok, yeah i'll add it to the spec i'm forming for the wheel modernisation23:37
*** dychen has joined #openstack-infra23:37
fungiianw: i can try to find time tomorrow to set up a temp server to test wheel builds on and see how many are not reproducible simply with a static SOURCE_DATE_EPOCH23:38
fungiand then that gives us a list of candidates to see what CFLAGS or other workarounds we might need to try23:39
*** armax has quit IRC23:41
*** dychen has quit IRC23:41
*** rfolco|brb is now known as rfolco23:41
*** dychen has joined #openstack-infra23:42
*** tosky has quit IRC23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!