Wednesday, 2019-01-30

*** ssbarnea|rover has quit IRC00:13
*** ssbarnea|rover has joined #zuul00:17
*** sdake has quit IRC00:19
*** saneax has quit IRC00:26
*** sdake has joined #zuul00:28
*** rlandy has quit IRC00:51
sean-k-mooneypabelanger: corvus: thanks for your help i tested https://review.openstack.org/#/c/633796/1 with my third party ci and it now works https://review.seanmooney.info/c/test/+/1041#message-9f4b6a8a00de3de14197c8a48466df27b141274801:14
sean-k-mooneypabelanger: i have also optimised my zuul config as you suggested to use a singel gerrit connection for all openstack related repos01:15
*** sdake has quit IRC01:19
*** sdake has joined #zuul01:23
pabelangersean-k-mooney: cool!02:36
*** sdake has quit IRC03:00
*** bhavikdbavishi has joined #zuul03:09
*** bhavikdbavishi has quit IRC03:38
*** bhavikdbavishi has joined #zuul03:44
*** chkumar|out is now known as chandankumar04:38
*** bhavikdbavishi has quit IRC05:37
*** bhavikdbavishi has joined #zuul05:44
*** bhavikdbavishi has quit IRC05:57
*** bhavikdbavishi has joined #zuul06:17
*** badboy has joined #zuul06:23
*** swest has joined #zuul06:30
*** quiquell|off is now known as quiquell06:39
*** jesusaur has quit IRC07:08
*** jesusaur has joined #zuul07:12
*** saneax has joined #zuul07:18
*** bhavikdbavishi has quit IRC08:03
*** gtema has joined #zuul08:10
*** pcaruana has joined #zuul08:11
*** avass has joined #zuul08:11
*** gtema has quit IRC08:14
*** gtema has joined #zuul08:18
openstackgerritFabien Boucher proposed openstack-infra/nodepool master: Add ip-pool option to the openstack provider  https://review.openstack.org/61952508:19
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/buildsets route  https://review.openstack.org/63003508:19
*** themroc has joined #zuul08:21
*** gtema has quit IRC08:24
*** gtema has joined #zuul08:38
*** bhavikdbavishi has joined #zuul08:41
quiquellHello08:44
quiquellIs there any way to share an openstack tenant with two nodepool-launchers ?08:44
quiquellSo they don't clean each other images I suppose it's an issue with quotas and the like08:44
*** electrofelix has joined #zuul08:51
*** jpena|off is now known as jpena08:57
*** saneax has quit IRC09:25
*** saneax has joined #zuul09:27
badboycan I use an ssh command on a nodepool worker node?10:04
badboyI would like to add a comment as a part of a playbook10:05
tobiashquiquell: I think if you name the providers of the config differently it might work10:12
quiquellsshnaidm: ^10:12
tobiashquiquell: but no guarantee about that ;)10:13
tobiashand quota management won't work10:13
quiquelltobiash: we don't care about that I think10:14
quiquelltobiash: ack will look thanks !10:14
tobiashbadboy: what do you mean exactly?10:14
tobiashbadboy: nodepool only checks ssh availability and retrieves host keys, it doesn't do a login or scripting on nodes10:15
badboytobiash: I was talking about Zuul10:15
tobiashbadboy: ansible by default uses ssh, what's your use case?10:16
tobiashbadboy: ah, you want to run ssh from the worker node to somewhere else?10:16
tobiashbadboy: that's easy with a shell task10:17
badboytobiash: yes10:17
tobiashyou might need to use secrets to add a private ssh key before though10:17
badboytobiash: I want to do something like this: ssh -p 29418 my-gerrit.example.com gerrit review -m $'"Some text"' <change_id>10:18
badboytobiash: the same user exists on the Zuul server and the worker node10:18
tobiashbadboy: yes, you can simply run this in a shell task if you prepared the private key before that10:18
badboytobiash: with the same key10:18
*** sshnaidm has quit IRC10:19
tobiashbadboy: you have two choices then: either add that key as a secret or mount that key into the job workspace using (un)trusted_rw_paths config of the executor10:20
tobiashI'd suggest the secret, because (un)trusted_rw/ro_paths are mounted into *every* job which is probably not what you want10:20
badboytobiash: the key is already in the ~/.ssh/id_rsa10:20
tobiashon the node?10:20
badboyyes10:21
tobiashok, then just run ssh in a shell task10:21
tobiashthen it's really easy10:21
tobiash(but also unsafe) ;)10:21
badboytobiash: I just want to get it all started10:23
badboytobiash: after that I will fix that10:23
badboytobiash: I keep getting Host key verification failed10:24
badboytobiash: on that worker-node10:25
badboytobiash: even though it's using the correct user and the keys are ok10:25
tobiashbadboy: host key verification is not about authentication10:25
tobiashyou need to either gather the host keys and put them into known_hosts or just ignore host key verification10:26
*** sshnaidm has joined #zuul10:26
tobiashbadboy: you can disable it in the ssh command line: ssh -o StrictHostKeyChecking=no10:27
quiquelltobiash: Has found a real weird stuff10:28
quiquelltobiash: looks like having a jinja stuff at git comment breaks zuul10:28
quiquelltobiash: https://review.rdoproject.org/r/#/c/18623/10:29
badboytobiash: thx, adding to known_hosts worked10:29
badboytobiash: one more, which zuul variable contains the commit sha?10:30
tobiashcheck out the inventory, it contains all variables10:35
tobiashbadboy: the vars are also documented here: https://zuul-ci.org/docs/zuul/user/jobs.html#zuul-variables10:39
badboytobiash: I am reading that atm and still cannot extract the change Id or commit id10:40
tobiashbadboy: change id and patchset are its own variables10:40
tobiashhere: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.items.change10:40
tobiashbadboy: but if you want zuul to report line based comments, check this out: https://zuul-ci.org/docs/zuul/user/jobs.html#leaving-file-comments10:41
tobiashin case that fits your needs, zuul itself will take care about reporting line based comments10:41
badboytobiash: so I should use {{ zuul.items[].change }} instead of {{ zuul.change }} ?10:42
tobiashbadboy: oh, you can use zuul.change and zuul.patchset10:44
tobiashlooks like these two are undocumented10:44
badboytobiash: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.change10:44
tobiashoh, overlooked that10:45
badboytobiash: http://paste.openstack.org/show/744244/10:46
badboytobiash: I would like to extract the Change-Id10:47
badboytobiash: currently it's stored as a part of the commit message10:47
tobiashthat's a gerrit thing, so you'll need some jinja/bash magic to extract it10:48
tobiashbut typically the gerrit api works with the change number and patchset10:48
tobiashso if your use case is to comment, you don't need the Change-Id from the commit message10:49
badboytobiash: ssh -p 29418 my-gerrit.example.com gerrit review -m $'"Some text" {{ zuul.change }},{{ zuul.patchset }}10:54
badboytobiash: ^ that's working except for the newline chars10:54
tobiashcool10:54
badboytobiash: thanks for your help10:56
tobiashyw10:58
*** bhavikdbavishi has quit IRC11:02
*** avass has quit IRC11:03
*** gtema has quit IRC11:20
electrofelixhughsaunders: are you still working on the nodepool agent plugin for Jenkins?11:31
*** gtema has joined #zuul11:48
tristanCquiquell: that's indeed a weird stuff, turns out the commit message ends up not escaped in the inventory and jinja expression are evaluated by template task :-)11:53
quiquelltristanC: weird...11:54
quiquelltristanC: crearly a bug, going to put one at story board11:54
quiquellwill try to debug11:54
*** luizbag has joined #zuul11:58
quiquelltristanC: I see we have the commit message in the invetory, is this new ?11:59
tristanCquiquell: you can debug the zuul.message variable, and for some reason it get "jinja" evaluated in any task using template11:59
quiquelltristanC: going to throw a review for this11:59
quiquelltristanC: I suppose inventory.yaml itself is a jinja template so weird stuff hapend like nested expansion12:00
*** sdake has joined #zuul12:02
tristanCthen we might want to escape jinja expressions before storing the message in the inventory...12:02
quiquelltristanC: Something like that I am looking at the code12:02
*** jpena is now known as jpena|lunch12:04
*** bhavikdbavishi has joined #zuul12:05
tristanCquiquell: it should be done somewhere like https://git.zuul-ci.org/cgit/zuul/tree/zuul/executor/client.py#n18112:09
quiquelltristanC: Yep I am reproducing it first with a unit test12:09
quiquellThen fix12:09
*** steddy has joined #zuul12:16
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory  https://review.openstack.org/63393012:30
steddyHi, I'm trying to debug a zuul installation.  I followed the 'Zuul from scratch' installation guide and have connected my install to github.  I'm trying to debug an issue where I recheck a pull request, then pipeline briefly flashes up in the status page but then dissappears with no logging. Can anyone help me get some debug logs to troubleshoot the issue.  Thanks12:36
*** sdake has quit IRC12:43
*** quiquell is now known as quiquell|lunch12:45
*** sdake has joined #zuul12:45
tobiashquiquell: does it fail to completely run the job containing jinja stuff in the commit message or does it just do weird stuff if you want to do something with it?12:46
tobiashah, looked at that link and everything is in retry limit12:47
*** rlandy has joined #zuul12:54
*** quiquell|lunch is now known as quiquell12:57
tristanCtobiash: it seems like zuul variables are evaluated for any task using template12:57
tobiashk12:58
*** jpena|lunch is now known as jpena13:02
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Make git clone timeout configurable  https://review.openstack.org/63393613:08
*** badboy has quit IRC13:14
*** gtema has quit IRC13:25
dmsimardFYI: ara 0.16.3 was just tagged and released to address a regression in the upcoming release of Ansible 2.813:31
*** bhavikdbavishi has quit IRC13:33
openstackgerritBenedikt Löffler proposed openstack-infra/zuul master: Report the build execute_time via mqtt  https://review.openstack.org/63394513:47
*** pcaruana has quit IRC13:50
openstackgerritSorin Sbarnea proposed openstack-infra/zuul-jobs master: Fix failure to add user to docker group on centos  https://review.openstack.org/63394813:54
*** pcaruana has joined #zuul13:57
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: WIP Revert "Revert "Add a timeout for the image build""  https://review.openstack.org/63379213:59
quiquelltobiash, tristanC: Maybe we can escape jinja stuff at https://github.com/openstack-infra/zuul/blob/master/zuul/executor/server.py#L588  ?14:06
tobiashquiquell: yes, I think that's the most generic place where you could do it14:08
quiquelltobiash: Will just do it for zuul.message14:08
tobiashquiquell: we should think about doing this for all values in that dict14:09
quiquelltobiash: this can have performance penalty14:09
tobiashbecause any jinja expression in that dict will brean this14:09
tobiashquiquell: how long do you expect this to take?14:09
quiquelltobiash: only for "zuul" herarchy ?14:09
quiquelltobiash: I see a lot of stuff here http://logs.rdoproject.org/21/18621/8/check/tripleo-ci-reproducer-fedora-28-libvirt/7c25ec9/zuul-info/inventory.yaml14:10
tobiashI guess escaping the zuul hierarchy will just take a few milliseconds14:10
quiquelltobiash: most of it is mechanical14:10
quiquelltobiash: the probability that it has jinja is low, but commit message having jinaj is high14:10
tobiashwell, escaping is just a few string operations right?14:11
tobiashso solving it in the whole zuul hierarchy prevents us from future occurrences of jinja stuff14:11
quiquelltobiash: let's escape all is more secure we can improve later on14:11
tobiashI'm pretty sure you won't be able to noteice any performance difference...14:12
tobiashthe git fetch and merge is probably thousands of times slower than the escaping ;)14:12
quiquelltobiash: what is the best unit test to test this ?14:13
quiquelltobiash: have try with the wrong one14:13
tobiashquiquell: I think you started good in test_inventory14:14
quiquelltobiash: ack will continue there14:14
tobiashof course you need to change the assert to expect the escaped value14:14
quiquelltobiash: yep was just playing around to understand that14:15
quiquelltobiash: going to scape {{ with {{ '{{' }} and the counter part14:18
quiquellis this good ?14:18
tobiashI think that will probably work14:18
*** sdake has quit IRC14:20
adam_ganyone using the github driver in production?14:27
pabelangeryes14:28
pabelangeransible-network / awx / ansible-runner are gatting with zuulv3 and github14:28
pabelangertobiash is also using GHE14:28
adam_gpabelanger: does it get very much traffic?14:28
pabelangeradam_g: not as much as tobiash, but a bit14:29
tobiashadam_g: we're using it to gate ~1000 repos14:29
tobiash(on ghe)14:30
tobiashand it didn't kill our github in the last year ;)14:31
adam_gah14:31
adam_gany stats on how many events per hour its typically handling?14:31
adam_gwe're running one against an internal GHE and find the queue grinds to a halt with any significant number of incoming events14:32
adam_ggranted its an older codebase (2.5 + the experimental github integration patches), but looking at master the issues im seeing should still be there14:32
tobiashI'd first have to request stats of our github14:32
adam_gwe started getting rate limited hard by our GHE, but its because they put a frontend LB in front that broke etag caching14:33
tobiashbut you should at least upgrade to ghe 2.12 which introduces github app auth (which I'd recommend)14:33
Shrewsoh lookie, adam_g does still exist  :)14:33
adam_gtobiash: interesting, ill check into that. we probably support that but dont use it14:34
adam_gi did find getChange() for GH is god awful slow, and is the source of our problem14:34
tobiashadam_g: when using github app auth, each installation has its own rate limit btw14:34
tobiashso when running a large zuul I'd strongly recommend gh app auth14:34
*** sdake has joined #zuul14:34
pabelanger+114:35
adam_gtobiash: whats the app auth buy wrt network round trips?14:35
adam_gi found shortcutting github3 a little bit sped things up quite a bit14:35
adam_gfrom ~48sec for 24 changes to 11sec14:36
adam_gShrews: o/14:36
tobiashadam_g: app auth doesn't change the number or kind of api calls14:37
adam_gevent queue processing takes many minutes while the run_handler lock is held and the wohle scheduler grinds to a halt trying to catch up14:37
*** bhavikdbavishi has joined #zuul14:38
adam_gi can imagine it gets much worse when getChange() is called recursively for Depends-on14:39
tobiashwell, I already had a year of optimization of the github driver ;)14:40
tobiashthe state of one year ago probably would kill our github now14:41
adam_gyea.. ive looked at git history trying to find key optimizations.  the problem isnt on the github side for us. enabling debug logging for requests/github3.py provided some alarming insight :)14:43
tobiashadam_g: btw, zuul hooks a caching layer into the requests session of github3.py14:44
adam_gyep14:45
adam_git still requires roundtrips for the conditional GET14:45
adam_gwhat ive seen is it churns for a loooong time gathering user permissions14:46
tobiashthat's true14:46
adam_ge.g., https://github.com/openstack-infra/zuul/blob/master/zuul/driver/github/githubconnection.py#L128314:47
*** quiquell is now known as quiquell|off14:48
adam_gi wish the people maintaining our zuul could get it upgraded to 3.0/master14:48
adam_gbut i will probably try to forward-port some of these patches in the meantime14:49
tobiashadam_g: you're on zuul2?14:49
*** sdake has quit IRC14:49
adam_gtobiash: yep. :(14:50
adam_ghttp://paste.openstack.org/show/744254/14:53
adam_gi havent tested but looking thru master it seems like those numbers would be similar14:54
*** sdake has joined #zuul14:54
adam_gwas mostly curious to know if the scheduler changed in some way where that slowness wouldnt effect other queue processing14:55
tobiashadam_g: most github api requests are done in the driver loop atm, so the queue processing is mostly unaffected14:55
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Escape jinja2 stuff from inventory  https://review.openstack.org/63393014:56
tobiashat least all api requests that do get change etc from events14:56
adam_gtobiash: oh interesting, which is the driver loop?14:59
*** themroc has quit IRC15:00
tobiashadam_g: https://git.zuul-ci.org/cgit/zuul/tree/zuul/driver/github/githubconnection.py#n42715:04
adam_gtobiash: oh, nice15:04
tobiashthat preprocesses the webhooks and inserts trigger events into the scheduler main loop15:05
adam_gcool. more ammo to throw at our CICD people to upgrade :)15:07
pabelangertobiash: corvus: clarkb: I wouldn't mind if we discussed maybe a zuul release this week, to help pick up recent github fix for issue we had in SF.io, along with gearman client keepalive.15:09
openstackgerritBenedikt Löffler proposed openstack-infra/zuul master: Fix flake8 error: E117 over-indented  https://review.openstack.org/63397415:10
*** sdake has quit IRC15:11
*** sdake has joined #zuul15:14
*** quiquell|off has quit IRC15:19
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Revert "Revert "Add a timeout for the image build""  https://review.openstack.org/63379215:25
Shrewscorvus: that ^ version seems to work as a daemon (i was getting log file output, at least)15:26
*** quiquell has joined #zuul15:33
*** quiquell is now known as quiquell|off15:33
Shrewsi wonder if we should try it first on nb03. we don't seem to have a real good way to test the builder as a daemon  :/15:40
tobiashpabelanger: sounds good to me, are there any known issues with the new containers?15:41
pabelangertobiash: I am unsure, I haven't been following along much on the container work15:43
*** arxcruz|ruck is now known as arxcruz15:58
*** pcaruana has quit IRC16:01
*** gtema has joined #zuul16:01
*** saneax has quit IRC16:01
*** luizbag has quit IRC16:03
*** steddy has quit IRC16:03
openstackgerritTobias Henkel proposed openstack-infra/zuul master: github: log the project name when getBranches fails  https://review.openstack.org/58384116:10
*** luizbag has joined #zuul16:10
tobiashtristanC: I've picked up this, I hope that's ok for you ^16:10
openstackgerritTobias Henkel proposed openstack-infra/zuul master: github: log the project name when getBranches fails  https://review.openstack.org/58384116:13
*** sdake has quit IRC16:14
*** pcaruana has joined #zuul16:17
*** ssbarnea|rover has quit IRC16:20
*** ssbarnea has joined #zuul16:21
*** sdake has joined #zuul16:22
openstackgerritMerged openstack-infra/zuul master: Fix flake8 error: E117 over-indented  https://review.openstack.org/63397416:39
*** pcaruana has quit IRC16:45
*** electrofelix has quit IRC16:49
*** sdake has quit IRC16:51
*** sdake has joined #zuul17:01
openstackgerritSorin Sbarnea proposed openstack-infra/zuul-jobs master: Fix failure to add user to docker group on centos  https://review.openstack.org/63394817:01
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP add provides/requires support  https://review.openstack.org/63360517:07
*** bhavikdbavishi has quit IRC17:19
*** bhavikdbavishi has joined #zuul17:19
*** gtema has quit IRC17:37
*** bhavikdbavishi has quit IRC17:58
*** saneax has joined #zuul18:10
*** jpena is now known as jpena|off18:13
*** pcaruana has joined #zuul18:17
*** bhavikdbavishi has joined #zuul18:19
*** themroc has joined #zuul18:23
*** saneax has quit IRC18:23
*** sshnaidm is now known as sshnaidm|afk18:25
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add provides/requires support  https://review.openstack.org/63360518:29
adam_gtobiash: so i was just looking closer at the driver loop you pointed me at. i dont think that really solves the issue ive been hitting. what im seeing is processing of the trigger events in the scheduler's trigger_event_queue taking so long that on a busy server it never manages to work thru the queue and never leaving here https://github.com/openstack-infra/zuul/blob/master/zuul/scheduler.py#L100918:38
SpamapSadam_g: moar CPU18:46
adam_gSpamapS: no, its *lots* of network round trips to GH for every change18:47
SpamapSadam_g: ah, I thought the GH bits happened before that loop.18:48
SpamapSadam_g: honestly, etag breakage sounds like your problem, not Zuul.18:48
SpamapSadam_g: but also the app model would probably help a lot since it might give you more rate limit pools18:49
adam_gSpamapS: its getChange() in the connection that does a buncha stuff.18:49
adam_gSpamapS: nah etags are actually working again as expected but still 1 require API round trip per for a hit, and double for miss :)18:50
adam_g*require 1 API RT18:50
adam_gthe one that really sucks is this https://github.com/openstack-infra/zuul/blob/master/zuul/driver/github/githubconnection.py#L114918:51
adam_gfor really active PRs especially18:51
SpamapSadam_g: I wonder if you could spin up a v3 and let it consume all the hooks for a while, see if it does better (maybe by multiplexing the hooks)18:51
SpamapSalso I wonder if you could actually benefit from a thread pool.18:52
SpamapSoh and yeah, that particular method caused us concern back in the Bonny days, I remember.18:53
adam_gSpamapS: id love to but dont have any cycles to do it, im already punting on other work to debug this since nobody else around here seems able to :)18:53
adam_gyep18:53
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add provides/requires support  https://review.openstack.org/63360519:12
*** luizbag has quit IRC19:17
*** panda is now known as panda|off19:59
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Revert "Revert "Add a timeout for the image build""  https://review.openstack.org/63379220:01
*** sdake has quit IRC20:06
*** sdake has joined #zuul20:06
*** bhavikdbavishi has quit IRC20:10
*** rfolco has quit IRC20:13
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Revert "Revert "Add a timeout for the image build""  https://review.openstack.org/63379220:20
*** rfolco has joined #zuul20:24
*** sdake has quit IRC20:39
*** rfolco has quit IRC20:40
*** sdake has joined #zuul20:41
*** sdake has quit IRC20:44
*** sdake has joined #zuul20:47
*** sdake has quit IRC21:01
*** sdake has joined #zuul21:02
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add provides/requires support  https://review.openstack.org/63360521:04
*** sdake has quit IRC21:38
*** sdake has joined #zuul21:49
*** openstackgerrit has quit IRC21:50
*** ianw_pto is now known as ianw21:59
*** openstackgerrit has joined #zuul22:07
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add provides/requires support  https://review.openstack.org/63360522:07
*** themroc has quit IRC22:31
corvustobiash, SpamapS: ^ that is green.  i know it *looks* like a big change, but there's actually only 189 lines of new code (46 of which is a sql migration).  the interesting bits are all in model.py.  the remaining 830 lines are testing, docs, and re-organization.  mostly testing.23:00
*** sdake has quit IRC23:01
*** rlandy is now known as rlandy|bbl23:24
*** sdake has joined #zuul23:29
*** sdake has quit IRC23:33
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Allow stage-output to copy root-owned files  https://review.openstack.org/63406923:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!