Monday, 2017-06-26

*** jkilpatr has quit IRC00:40
jlkholy backscroll04:25
*** hashar has joined #zuul07:48
*** smyers has quit IRC09:04
*** hashar has quit IRC09:13
*** hashar has joined #zuul09:14
*** smyers has joined #zuul09:15
*** jkilpatr has joined #zuul11:06
mordredharlowja, Shrews: I mean - we ARE python3 only, so we could just use Barrier if it's the right thing to use11:39
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Fix dict key copy operation  https://review.openstack.org/47690211:39
Shrewsmordred: nodepool is not11:39
Shrewsunless we remove py2 support from nodepool too11:40
mordredShrews: no reason to keep py2 support for nodepool zuulv3 - especially if there's a useful primitive in python3 that's not in 211:41
mordredbut with zuulv3 being py3 only, I don't know that there's much value in keeping python2 support in nodepool, right?11:41
Shrewsi dunno. people like running nodepool w/o zuul too, or so i hear11:42
Shrewsbut i'm all for removing py211:42
mordredyah - definitely agree about people running nodepool w/o zuul11:43
mordredbut I think the logical argument is still the same - nodepool v3 is a whole new thing so we really shouldn't have any 'legacy' installs that would be screwed - might as well do it now before we grow any11:43
mordredwe might want to get infra's v3 nodepool running on v3 though - I think we rolled it out on v211:44
Shrewsmordred: complication with that is we run v3 nodepool-builder in production on py2, so tough to get rid of right now12:04
Shrewsi mean we still could, but would require quite a bit of work, i would think12:05
mordredShrews: yah - I think we need to migrate that to runningin production on py3 first12:05
mordredbut I think we should do it, since we're requiring py3 for zuul itself, at least before we officially release v3 - otherwise we'll be in a pretty odd state :)12:06
Shrewsmordred: i'll put it on the agenda for discussion12:06
mordredwoot12:07
*** dkranz has joined #zuul12:13
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Abstract Nodepool request handling code  https://review.openstack.org/46874812:43
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Abstract Nodepool provider management code  https://review.openstack.org/46874912:43
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Collect request handling implementation in an OpenStack driver  https://review.openstack.org/46875012:43
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Extend Nodepool configuration syntax to support multiple drivers  https://review.openstack.org/46875112:43
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool  https://review.openstack.org/46862412:43
*** jkilpatr has quit IRC12:54
tobiashSpamapS: had another hang12:55
tobiashpost mortem strace of ansible-playbook is:12:55
tobiashfutex(0x557001244fc0, FUTEX_WAIT_PRIVATE, 4294967295, NULL12:55
tobiashwill probably have to install a python debugger in the zuul container to be able to get a python stacktrace12:58
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add tox py35 environment  https://review.openstack.org/47752512:59
Shrewsugh, ignore that13:03
*** isaacb has joined #zuul13:04
Shrewsharlowja: it would probably make more sense to propose https://review.openstack.org/476683 on the feature/zuulv3 branch rather than master. It would make removing py2 support easier.13:08
*** jkilpatr has joined #zuul13:08
mordred++13:09
SpamapStobiash: ugh that means it's hung on "some pythony thing" :-P14:02
mordredSpamapS, tobiash: maybe the gdb python stuff jeblair did a couple of weeks ago will be useful14:18
dmsimardpretend I want to pass env variables to a ready-script (sorry, not upstream nodepool version), how would you do it ?14:51
dmsimardproblem I have is getting configure_mirror.sh ( https://github.com/openstack-infra/project-config/blob/master/nodepool/scripts/configure_mirror.sh ) to pick up a specific NODEPOOL_MIRROR_HOST and can't for the life of me figure it out14:52
dmsimardI tried inserting an element that persists the env var I want in /etc/profile -- I can log in as jenkins and see the env var but nodepool/configure_mirror.sh doesn't seem to pick it up (maybe it runs the script with sudo without -E ?)14:53
dmsimardexample error output from nodepool booting the node: http://paste.openstack.org/show/613729/14:53
dmsimardpabelanger, mordred ^ ?14:53
dmsimardAt this point what I'm considering is to add a file to source in configure_mirror.sh (say, /etc/nodepool/extra-vars.sh or something), source it if it exists and then configure an element to write the things I need in that file14:55
clarkbdmsimard: it is supposed to be based on the data coming out of /etc/nodepool. I guess you don't want it to be cloud/region specific but based on some other info?14:56
dmsimardclarkb: yeah, I'm basically trying to build images near-identical to upstream but we don't run mirrors in the region that configure_mirror is expecting.14:57
clarkbI would use a different ready script then14:58
mordreddmsimard: I'd just patch your configure_mirror script (or ready script) to hard-code your mirror14:58
dmsimardmordred, clarkb: we're using pure-upstream elements and scripts from project-config/nodepool so using a different ready script is less than ideal14:59
dmsimardwe're not forking project-config, we're replicating it and leveraging it as-is -- the goal being to stay as close to upstream as possible15:00
dmsimardi.e, I don't want to maintain a fork and the people interested in these jobs want the images to be the same-ish as upstream15:00
dmsimardthe image definition is here, there are only very subtle differences with the upstream one: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/nodepool/nodepool.yaml#L12-L3415:02
dmsimardFWIW the NODEPOOL_MIRROR_HOST env var in that image definition is for test purposes for the time being15:03
clarkbdmsimard: the script should allow each of the vars to be overridden, I would wrap it with your own script that overrides the vars15:03
dmsimardclarkb: ok, I'll try that approach15:04
mordreddmsimard: I know you're not there yet - but just to re-iterate, the issues you're running in to are one of the main drivers behind the v3 work - so when you get free time, I'd love if you could give feedback on whether or not it's still a problem in v3 (not urgent, I know you have a ton of work on your plate - but the earlier we can verify you're good or not the better, right?)15:07
dmsimardmordred: yeah this kind of stuff wasn't obvious to me before -- we're hoping software factory to get closer to upstream asap15:10
dmsimardWhile perhaps blasphemous in these lands, I'm generally frustrated with DIB. I don't like it at all. :(15:11
clarkbNODEPOOL_MIRROR_HOST=${NODEPOOL_MIRROR_HOST:-mirror.$NODEPOOL_REGION.$NODEPOOL_CLOUD.openstack.org} confirmed, so you just have to set that value in a wrapping script and done15:13
dmsimardclarkb: yeah, that's what I'm trying right now -- I thought that persisting "export NODEPOOL_MIRROR_HOST=mirror" in /etc/profile would work but it doesn't.15:14
clarkbre dib, what is it doing that is problematic?15:15
dmsimardclarkb: oh, probably nothing. I just don't like the tool :)15:15
dmsimardclarkb: likely in part for the same reasons I dislike Dockerfiles, in the end it's just bash -- DIB is *layers* of bash :)15:16
dmsimardI like bash, but I just think it's not the right tool for that kind of job15:16
clarkbdmsimard: its actually layers of whatever you want. Happens that people like bash for interacting with systems15:16
dmsimardclarkb: right, I half forgot about that, there's ansible elements and stuff too now.15:17
clarkbdmsimard: its run parts of executable whatever. Could be emacs lisp for all dib cares15:17
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add configuration documentation  https://review.openstack.org/46332815:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Reorganize docs into user/admin guide  https://review.openstack.org/47592815:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use oslosphinx theme  https://review.openstack.org/47758515:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move status_expiry to webapp section  https://review.openstack.org/47758615:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move tenant_config option to scheduler section  https://review.openstack.org/47758715:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct sample zuul.conf  https://review.openstack.org/47758815:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use executor section of zuul.conf for executor dirs  https://review.openstack.org/47758915:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use scheduler-specific log config and pidfile  https://review.openstack.org/47759015:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move zookeeper_hosts to zookeeper section  https://review.openstack.org/47759115:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some information about canonical_hostname  https://review.openstack.org/47759215:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix some inconsistent indentation in docs  https://review.openstack.org/47759315:40
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename git_host to server in github driver  https://review.openstack.org/47759415:40
* jeblair deplanes15:40
jeblairtobiash: do you have information about your non-deterministic setup bug collected somewhere so i can try to understand the issue?15:43
*** isaacb has quit IRC15:57
jeblairtobiash: you said: "the playbook repos don't seem to be part of the repo state the executor gets" and "the playbook repos are also not part of the projects dict"16:04
jeblairtobiash: repo_state holds information on all the projects that are in the dependency chain for an item; so a playbook repo would only show up there if the current change depends on a change to that repo16:05
jeblairtobiash: that is used by the merger to help construct the same state for all of the repos with changes.16:09
jeblairtobiash: the projects key holds all of the projects which the job says that it requires (via job.required_projects).  also, all the projects in the dependency chain are added to it as well.  to be honest, i'm not sure we need that last part, other than just adding the project for the change being tested.16:09
jeblairtobiash: at any rate, that's used to prepare all of the repos in the work/src directory for the job to use16:10
jeblairtobiash: so if a playbook repo isn't a required-project for a job, and it doesn't show up in a dependency chain for the change, then it won't appear in either of those16:11
jeblairtobiash: in that case, we should just check out the branch-tip of the playbook repo16:11
jeblairtobiash: if the branch-tip changes (ie, a change to the playbook repo merges while running unrelated jobs) then what we check out would be non-deterministic -- we might check out the old version or the new version depending on exactly when the job ran.  i'm not certain that's a bug or something that we need to work around.16:13
jeblairtobiash: if, however, you get something different *without* any changes landing in the playbook repo, then we may have a bug such as using an unclean git tree to set up the playbook repo.16:14
*** isaacb has joined #zuul16:15
SpamapSclarkb: to give dmsimard some support.. dib is indeed "whatever you want", except most elements that exist already, are bash. So it ends up being layers of bash to run mostly bash anyway.16:23
SpamapSThat said.. I've not seen many build systems that don't end up working mostly the same.16:24
SpamapSonce you take runtime concerns out and just want to sequence things and explode when they break.. bash looks pretty good.16:24
clarkbwhile thats true it is a common misconception that it must be bash and dib itself is rewriting chunks of stuff in python aiui16:25
SpamapSAgreed and that last bit is good to hear.16:26
SpamapSI've found that fretting about the elegance of build-level tooling is a trap. The point of separating build from run is that build is linear and happens in a controlled environment.16:27
jeblairdmsimard: not sure if you got what you needed earlier, but for ready scripts, any environment variable in nodepool's environment that starts with "NODEPOOL_" will be passed through to the ready script16:28
jeblairdmsimard: ie, if you start nodepoold with "NODEPOOL_FOO", NODEPOOL_FOO will be available within the ready script16:29
dmsimardjeblair: oh, I didn't know that -- interesting.16:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix some inconsistent indentation in docs  https://review.openstack.org/47759317:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some information about canonical_hostname  https://review.openstack.org/47759217:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename git_host to server in github driver  https://review.openstack.org/47759417:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move tenant_config option to scheduler section  https://review.openstack.org/47758717:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use executor section of zuul.conf for executor dirs  https://review.openstack.org/47758917:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct sample zuul.conf  https://review.openstack.org/47758817:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move zookeeper_hosts to zookeeper section  https://review.openstack.org/47759117:00
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use scheduler-specific log config and pidfile  https://review.openstack.org/47759017:00
*** isaacb has quit IRC17:04
jlko/17:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move tenant_config option to scheduler section  https://review.openstack.org/47758717:52
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix some inconsistent indentation in docs  https://review.openstack.org/47759317:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some information about canonical_hostname  https://review.openstack.org/47759217:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename git_host to server in github driver  https://review.openstack.org/47759417:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use executor section of zuul.conf for executor dirs  https://review.openstack.org/47758917:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Correct sample zuul.conf  https://review.openstack.org/47758817:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move zookeeper_hosts to zookeeper section  https://review.openstack.org/47759117:53
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use scheduler-specific log config and pidfile  https://review.openstack.org/47759017:53
clarkbjeblair: https://review.openstack.org/476678 and https://review.openstack.org/476223 are two nodepool related items that came up last week18:01
clarkbI separately merged a fix that shrews had for master as well so we should consider restarting nodepool this week18:02
clarkbmordred: ^ how does that line up with shade releases18:02
jeblairclarkb: i guess a rainy day task would be to make uploads interruptable in shade?  would obviously require some significant work thoug.h18:03
jeblairclarkb: what's the master branch change you referred to?18:03
clarkbya, I don't think it would be very straightforward so probably not worth fixing immediately. Cna't hurt to support it though18:04
clarkbjeblair: you mean why did https://review.openstack.org/476223 come about?18:04
jeblairclarkb: "I separately merged a fix that shrews had for master" <-- that one18:05
clarkboh let me dig it up18:05
clarkbhttps://review.openstack.org/#/c/470364/ that one18:05
clarkbsorry shrews one is unrelated to the other two. It is just another fix that came  up so makes restarts worthwhile if shade is no longer unhappy18:06
jeblairclarkb: ack, thx18:06
SpamapSI'd have to look but I feel like Shade relies a lot on some specialness in swiftclient if uploading to swift. Dunno about glance direct PUT's.18:09
* SpamapS hasn't been tracking the de-client-ification effort very closely either18:10
jeblairSpamapS: the swiftclient stuff has been removed18:10
jeblairSpamapS: i think ironicclient is the only one left18:11
SpamapSyowza, that's nice18:11
jeblairya18:11
clarkbjeblair: re 476223 we have a few thousand failed image upload records now because requests decided to stop bundling certs in its repo and instead depends on certifi18:13
clarkb(this lead to running python processes not being able to load the certs file out of the requeists install location at runtime which failed all the ssl checking)18:13
jeblairclarkb: thanks18:15
tobiashjeblair: regarding non-deterministic playbook setup18:20
tobiashmy scenario is I have multiple mergers, one executor18:20
jeblairtobiash: aha! i think i see the problem18:21
tobiashI make a change to a config repo, that gets merged (currently with a noop job) so the executor doesn't necessarily update the repo18:21
tobiashwhen merged, possibly a merger gets the repo update job -> executor has old copy cached18:22
jeblairtobiash: because the playbook repo doesn't appear in 'projects', the executor doesn't tell its internal merger to update that repo, so it doesn't get the new branch tip when you do that ^18:22
tobiashexactly18:22
tobiashthat's why I thought it would be good to freeze the playbook repos also like the dependent changes18:23
tobiashthat would also make this independent of concurrent merges and also possibly reduce load on gerrit as the executor doesn;t have to do a fetch on every playbook repo if it has already the correct revision cached18:24
tobiashmy current workaround is to stop the mergers and restart the scheduler after a config repo change has landed ;)18:25
jeblairtobiash: it's difficult to put anything other than dependent changes into repo_state -- because we don't actually know what those repos should be until after that merge completes (it could contain configuration changes which add required-projects, or change the job config to alter which playbook repos are involved)18:25
tobiashmaybe the repo state could be appended in a second stage18:26
jeblairtobiash: the way to fix that which at least matches the rest of the current behavior would be to alter the executor to run an update task on all the playbook repos (and roles!) in addition to what's in 'projects'.18:26
tobiashyes, that would fix my problem18:27
jeblairtobiash: i think that executor change should be easy to do, so we should probably start there.  i think your idea of a second-stage freeze of related repos from the merger is worth considering, however, it could potentially slow operations.  we might mitigate it by asking for all of the known playbook and role repos in the first stage, and then we only have to do a second stage if something changes....18:28
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use executor section of zuul.conf for executor dirs  https://review.openstack.org/47758918:29
tobiashjeblair: sounds like a plan18:29
tobiashjeblair: another issue I discovered today (but had no time yet to debug):18:29
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Use scheduler-specific log config and pidfile  https://review.openstack.org/47759018:30
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix some inconsistent indentation in docs  https://review.openstack.org/47759318:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add some information about canonical_hostname  https://review.openstack.org/47759218:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Rename git_host to server in github driver  https://review.openstack.org/47759418:31
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Move zookeeper_hosts to zookeeper section  https://review.openstack.org/47759118:31
tobiashscenario: change A,1 runs a (longer running) job, during that A,2 adding or removing jobs in .zuul.yaml is pushed18:31
tobiashmy expectation is that A is restarted with A,2 with the jobs added/removed18:32
tobiashobserved behaviour is jobs of A,1 are restarted without honoring added/removed jobs18:32
jeblairtobiash: by 'pushed' do you mean 'proposed'?  or directly pushed onto the branch?18:32
tobiashproposed18:32
jeblairtobiash: (or do you mean merged?)18:33
tobiashfurther reporting is against A,118:33
tobiashno merge involved18:33
jeblairtobiash: okay, yeah that sounds like a bug :(18:33
jeblairtobiash: we have no unit tests for that case, but should be easy to add one.18:34
tobiashso this could be one or two bugs: job graph is not recalculated and reporting is against wrong patch version18:34
jeblairagreed18:34
tobiashthat was on my task list, but next two days I'm on the german openstack days and after that on a full day workshop18:35
tobiash-> three days off18:35
tobiashand I'll have to debug some occasional freeze of ansible-playbook (see scrollback)18:36
jeblairtobiash: cool, i might be able to take a look at it before then -- not sure how this week is going to go yet :)18:36
jeblairtobiash: that's unfortunate.  does zuul at least time out the job and kill the process correctly?18:36
tobiashdidn't wait for timeout but I think it would18:36
tobiashansible-playbook is stuck in some futex (before launching an ssh process)18:37
tobiashit might possibly relate to the modules we inject (log streaming?) as I didn't observe something like that running ansible manually18:38
tobiashlast log message before the freeze is "Starting to log 396bf1da93ed43adbfc03274c24a55ee for task..."18:40
tobiashbut I have to install gdb in the zuul container and try to get a stack trace next time it happend18:40
tobiashhappens in about 1-5% of runs for me18:41
tobiashwhat's strange about that is if I kill the stuck ansible-playbook process zuul restarts this playbook and continues the job if nothing happened18:42
tobiash(and succeeds the job)18:42
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Make sure playbook and role repos are updated  https://review.openstack.org/47762718:46
jeblairtobiash: ^ there's the fix for the first thing18:46
tobiashjeblair: crazy you're fast18:47
jeblairtobiash: i refreshed my memory of the code this morning :)18:48
tobiashjeblair: lgtm18:49
tobiashjeblair: regarding gerrit 2.13, I think it would be good to change the label matching stuff to case sensitive18:51
tobiashjeblair: gerrit 2.13 breaks the gate with wrong casing as vote+submit doesn't work with one command anymore if the label casing is not correct18:52
tobiashjeblair: https://review.openstack.org/#/c/469946/ that's my proposal18:52
jeblairtobiash: do you know if they did that on purpose?18:53
tobiash(as this breaks *every* fixture the tests are updated in a follow up change for easier review, but would need a rebase)18:53
jeblair(this seems like a bug, if setting the vote works but not submit)18:53
tobiashjeblair: I don't know, we just updated gerrit in march and that broke out gate18:53
jlkSpamapS: you rightly called me out on not having a test case to limit job reporting for push events in github. I implemented the test, and found more bugs.  https://review.openstack.org/47640018:55
jeblairtobiash: in 469946, should we remove the .lower() in normalize_category?18:55
tobiashjeblair: we could to be really correct, but I left this at first as this is just used for internal comparisions18:56
tobiashin this case I think it might be correct to remove the nodmalize_category completely18:57
clarkbre 2.13 we are getting much closer to being able to deploy it on our test box, review-dev.openstack.org so getting those issues fixed will shortly become much more important for us too18:57
jeblairclarkb: is that gerrit 2.13 issue on your radar ^?18:58
clarkbya I saw it last week I think was just too busy actually sorting out 2.13 and dns problems to pay close attention18:58
clarkbthe 2.13 upgrade is :(18:59
jeblairtobiash: well, the intent was that you would be able to say "code-review: 2" in the config file rather than '"Code Reveiew": 2'... if we remove the lower but leave the space/- translation, we can at least still say "Code-Review: 2"18:59
clarkbmanual db schema updates and migration from h2 to mysql19:00
jeblairclarkb: 2.11 put some stuff in h2?19:00
tobiashjeblair: ah, ok, but you would then spell the labels in requirements different than in the reporting?19:01
clarkbjeblair: no 2.13 did then in latest release they must've realized that was silly so if you upgrade to 2.13.7 first you have to manually do db things to go to 2.13.819:01
clarkbjeblair: so we reverted the 2.13.7 upgrade and will work out how to go straight to 2.13.8 to avoid some of the mess19:01
jeblairclarkb: we upgraded something to 2.13?19:01
clarkbjeblair: review-dev, for about a grand total of 10 minutes19:02
jeblairclarkb: ok -- is that back at its 2.11 state?19:02
clarkbyup we rolled back so that we can do 2.11 -> 2.13.8 testing accurately19:03
jeblairok cool.19:03
tobiashjeblair: so in case we want to keep normalize_category I would vote for leaving the tolower here as we anyway do a translation for internal comparison19:03
tobiashjeblair: but it's your decision, it should work with both19:03
jeblairtobiash: i think the thing i worry about is that mostly your change is making sure that we don't enqueue something that gerrit won't handle later...19:04
tobiashjeblair: landing such a change will have to be coordinated wisely as it will break *every* change (and also getx broken with every change)19:05
tobiashjeblair: so the part in gerritconnection makes sure gerrit can handle the vote+submit19:06
jeblairtobiash: i think i see why it's okay to leave the .lower() call in there.... it's because it's only the allow_needs that could actually cause a problem19:06
jeblairtobiash: even if we "incorrectly" decide that a change meets a pipeline requirement despite the cases being different, it won't be a problem unless it also shows up in an allow-needs...19:07
tobiashjeblair: yepp19:07
jeblairlet's leave it now; i'll think more about changing it later19:08
tobiashok19:08
jeblairtobiash: we should get some more eyes on that and merge it soon19:09
tobiashjeblair: ok, I'll rebase the test fix follow up once more eyes are happy with this19:09
tobiashrebasing the test fixtures is like hell ;)19:10
tobiashjeblair: I've a question about secrets19:11
tobiashas I've understood secrets are also allowed in untrusted repos19:11
tobiashhow are speculative changes to the playbooks handled in that case?19:11
jeblairaha!  missing documentation :)19:12
tobiashor is it just a risk to place secrets in untrusted repos?19:12
jeblairtobiash: pipelines have an allow-secrets attribute, so you can ensure that secrets aren't available to jobs running in pipelines which run unreviewed code19:13
jeblairtobiash: allow-secrets is false by default19:13
jeblairtobiash: so you probably only want to enable it for gate, or post, or some pipeline that requires at least a trusted code review before running jobs19:14
tobiashjeblair: I would have a use case as also some jobs in check need to download stuff from (authenticated) artifactory19:15
jeblairtobiash: (for example, we might make a pipeline for running shade jobs against real clouds using real creds which requires a shade-core to +2 the change before jobs run)19:15
SpamapSjlk: \o/ for testing19:15
* SpamapS will hit the review queue after lunch19:15
tobiashthe initial job inheritance structure is not retained during job freezing right?19:16
tobiashso currently it would not be possible to have a post playbook in a config repos using a secret for logs, artifacts, etc upload?19:17
tobiash(and hiding the secret from the main playbook)19:18
jeblairtobiash: secrets are available to the entire job.  it's starting to look like we may want to think about making them per-playbook, however we hadn't assumed we could maintain security in the executor environment between playbook invocations.  with bubblewrap for trusted jobs, it's starting to look like we might be able to do that.19:19
jeblairtobiash: but that's a moderate redesign, so i don't want to block on it right now.19:19
tobiashjeblair: so a task for 3.1?19:20
jeblairtobiash: maybe?  possibly still v3.0, but just later, if we decide secrets aren't useful without it.  :)19:21
jeblairtobiash: we've already run in to one place where we would like to use secrets but can't because of this issue.  we can work around it, but it's not ideal.19:21
jeblairso i'd like to keep an eye on it while we continue to try to get more jobs working to see what works and what doesn't.19:22
tobiashyes, so for my use case I would have to add the secret to artifactory the check pipeline and risk disclosure19:22
tobiashthis could be reduced with a dedicated secret for exactly this purpose19:23
jeblair(at least if we make the change, we'd be making the behavior *more* restrictive, rather than less, so it should be okay to make the change in that direction)19:23
jeblairtobiash: but you'd want to put that only in a trusted pre-playbook or something, right?19:23
tobiashjeblair: that was the plan19:23
tobiashtrusted pre or post playbook for downloads and uploads19:24
jeblairtobiash: gotcha.  that's similar to the thing we ran into as well, where we wanted to use a secret for log publishing.19:24
tobiashthat's my second use case19:24
jeblairtobiash: our plan to work around that is to have the log publishing key on the executor, so the trusted post-playbook can use it "out-of-band" so to speak19:25
jeblairtobiash: you may be able to work around your issue similarly19:25
tobiashjeblair: I'm thinking about using eventualley artifactory for log publishing. That way I would remove the need of backups in the ci system itself19:25
jeblairi'll start thinking about per-playbook secrets in the background19:26
tobiashjeblair: ok, that will work as untrusted jobs won't be able to read stuff (other than the workdir) from the executor?19:26
jeblairtobiash: yes -- and even with trusted bubblewrap, we should be able to map in specific directories to trusted playbooks, so you can map in your artifactory creds that way.19:27
tobiashjeblair: I read somewhere that there is an idea of site local variables19:27
tobiashjeblair: maybe that could be enhanced with site local trusted variables19:27
jeblairtobiash: https://review.openstack.org/44773419:28
Shrewsmordred: I'm *this* close to abandoning autobahn19:28
tobiashjeblair: ah, there I saw that :)19:29
jeblairtobiash: could be.  though this is looking like a defect in secrets, so if the only reason to do that is this, it may be better to improve secrets.  :)19:29
tobiash;)19:29
openstackgerritDavid Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add web-based console log streaming  https://review.openstack.org/46335319:30
jeblairtobiash: (i worry that encouraging folks to put sensitive data in site-wide ansible variables might lead to accidentally leaking them through jobs that weren't written with them in mind)19:30
tobiashjeblair: ok, so the workaround is to have a folder with secrets on the executor and having ansible resolving them with lookup (which doesn't work in untrusted jobs)19:30
*** ChanServ has quit IRC19:30
jeblairtobiash: yep19:30
jeblairShrews: i'm going to grab lunch now, but when i'm back, i'd like to hear more about the issues you're running into if you have a moment19:31
tobiashjeblair: agreed, having them somewhere in a folder is much better than publicly visible (in any trusted job) with a red flag *here are the secrets* on it19:32
Shrewsjeblair: i'm constructing a very defeated email  :)19:32
*** ChanServ has joined #zuul19:35
*** card.freenode.net sets mode: +o ChanServ19:35
tobiashjeblair: what I noticed in my experiments with job config with zuul managed roles import19:36
tobiashjeblair: I have a repo with a /playbooks folder, some jobs defined in .zuul.yaml and a role in /roles19:37
tobiashthese roles are to be used in the same repo as well as in other repos19:37
tobiashis it intended that I also have to define the roles property of jobs in the same repo?19:38
mordredShrews: yah?19:39
tobiashjeblair: like this: http://paste.openstack.org/show/613752/19:41
*** jkilpatr has quit IRC20:00
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix encrypt_secret for python3  https://review.openstack.org/47716820:01
jeblairmordred: do you have thoughts on tobiash's issue above?20:08
jeblairtobiash, mordred: should we add every playbook repo to the roles path as well?20:08
mordredjeblair: pondering20:09
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Fix encrypt_secret for python3  https://review.openstack.org/47716820:10
tobiashjeblair: that would simplify job config20:11
tobiashmy use case is to provide some public zuul managed roles and in the same repos having jobs to test them20:11
mordredjeblair: there is a somewhat related question from a couple of weeks ago about20:11
mordredgah20:11
tobiashcurrently I have to mention the repo itself in the job description which has two drawbacks: first, visual noise, second probably more important, the repo is not relocatable20:13
tobiash(without fixing the jobs referring to its own repo)20:13
jeblairyeah, i feel like "playbook should be able to use roles in its own repo" seems like a fairly normal expectation....20:14
jeblairtobiash, mordred: if the "roles/" directory were under "playbooks/" would this automatically work?20:14
mordredyah. I'm trying to come up with drawbacks20:14
mordredjeblair: yes20:14
tobiashjeblair: that works20:14
mordredjeblair: but then the dir would be weird for folks consuming it as a source of roles20:15
mordreds/dir/repo/20:15
jeblairright -- though, having roles as a sibling dir to a playbook is the more normal ansible thing...20:15
tobiashroles in playbooks/roles are repo private currently20:15
jeblairthis could also be solved with a symlink, i bet...20:16
jeblairor we could move the playbook up to the root level20:16
tobiashjeblair: right that would also work20:16
tobiashbut I like the idea of public roles in /roles and playbook private roles in /playbooks/roles20:16
mordredthe thing the repo is providing is the content in roles/ - the playbooks dir, in this case, exists to test the roles20:17
tobiashjepp20:17
jeblairthen that leaves us with (a) auto-add playbook repos as roles or (b) symlink20:17
mordredyah20:17
mordredI can't come up with any immediately bad reasons to not do (a)20:17
tobiashI'd vote for a20:18
jeblair(b) is more explicit... it lets you indicate what's going on for this specific repo20:18
jeblairbut yeah, (a) may be sort of 'just works'20:18
mordredjeblair: while we're considering this topic ...20:19
mordredjeblair: it might be worth noodling on repos in the 'standard' role layout, such as: http://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/tree/20:20
jeblairmordred: whatcha mean?20:21
tobiashmordred: isn't that supported (I think it was named bare role in the docs)?20:21
mordredjeblair: which put playbooks to test the role in question in the tests/ dir20:21
tobiashmordred: ah, configurable playbooks dir?20:21
tobiash:)20:21
jeblairmordred, tobiash: yeah that roles layout is supported (yes, we call it a bare-role).  and i think we've had a few requests for not assuming playbooks are in playbooks/20:23
mordredjeblair: ok. cool20:23
jeblairmordred: is that the suggestion?  that we make it so that we can have a job run 'tests/test.yml' as the main job playbook?20:23
jeblairmordred: or were you more focused on ensuring that we support bare-roles?20:23
jeblairhttp://docs-draft.openstack.org/27/477627/1/check/gate-zuul-docs-ubuntu-xenial/2a62ff4//doc/build/html/user/config.html#job20:24
jeblair^ describes bare-roles; and i just realized we're going to need deep links to the keywords there and we don't have them :|20:24
mordredjeblair: oh - no - I was more focused on making sure that folks with bare-repo ansible roles can test them in an easy manner20:24
mordredjeblair: so I think that would be the playbooks/ topic20:25
jeblairmordred: yeah, so on that, i was thinking we keep the implicit playbook name under playbooks, but then if you set "run:" then we drop the playbooks/ prefix and expect you to pass tho whole path.20:26
jeblairthat's like a one-liner.20:26
mordredjeblair: ++ wfm20:26
tobiash++20:26
mordredjeblair: for the other, the main drawback I can think of to the symlink approach for the tobiash role thing is that it would prevent him having additional private roles that are just for testing20:26
mordrednow - how likely  _that_ is is another question20:27
jeblairmordred: on the bare-role topic -- we support either "bare" roles (like your example) or "contained" roles like tobiash's as targets for the "roles" job parameter....20:27
jeblairmordred: there is a complication if we want to extend that to plan (a) [automatically add playbook repos as roles]20:28
mordredjeblair: how hard would it be to add playbook repos if they have a roles dir?20:28
jeblairmordred: it's easy to add "contained" roles to roles_path.  however, we need a name for the "bare" roles case.20:28
jeblairmordred: we can just use the repo name20:28
mordredjeblair: OH - yah - I see the additional issue20:29
jeblairmordred: and then, if someone needs the role to have a different name than the repo, you have to add an explicit "roles" entry for it.20:29
jeblairmordred: (with a roles entry, you can rename the role to something other than the repo name -- eg, "openstack-ansible-os_nova" to "os_nova")20:29
mordredyah, if you look at: http://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/tests20:31
mordredthat's an issue :)20:31
jeblairmordred: role: "{{ rolename }}"  ?20:32
mordredjeblair: oh - fwiw, there is an ansible "standard" which is that repo names starting with "ansibe-role-foo" install as foo20:32
mordredjeblair: the test job in that case passes in rolename20:32
mordredhttp://git.openstack.org/cgit/openstack-infra/ansible-role-puppet/tree/run_tests.sh20:32
jeblairmordred: so that would run with rolename=ansible-role-puppet?20:33
jeblairmordred: if so, then adding that playbook repo as a role (plan a) would just work, i think.20:34
mordredjeblair: yes. however, it's always felt a little hacky to me- what I'd prefer is to have the playbook not take a rolename argument and to be written as roles: - puppet (since that's the role name) - but it's harder to run it that way locally since that's not the dir it'll be cloned in to20:35
mordredhang on - reading a thing real quick20:36
tobiashpaths: jeblair, mordred: dropping playbooks/ when you set run: could be inconsistent with pre and post playbooks as they are set in any case20:37
tobiashpaths: jeblair, mordred: maybe we could distinguish the path if it's an absolute path (starting with /) or not20:37
mordredjeblair, tobiash: this is a role from geerlingguy - and his approach is the top google hit for "testing ansible roles" https://github.com/geerlingguy/ansible-role-java/blob/master/tests/test.yml20:37
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Fix parsing of github PR url  https://review.openstack.org/47628620:38
mordredjeblair: I think we can do better than that, tbh20:38
jeblairmordred: we could have the executor be smarter and automatically rename ansible-role-foo to foo for the role install (but not, obviously for the main project checkout).  that would make your 'puppet' case work, but break that test as written.  maybe that's desirable.  :)20:39
mordredjeblair: yah - if I can get zuul testing with that break, I'll take it :)20:40
jeblairtobiash: i think we could drop automatic playbooks/ for pre and post as well.  so it's only automatic for the implied job playbook.20:40
tobiashjeblair: so pre/post playbooks must be defined absolute then?20:41
mordredjeblair: I think dealing with ansible-role- prefix would be a nice nod to compat with galaxy20:41
jeblairtobiash: or your suggestion for matching on '/' would work, if we want to try to support both.  i guess it would be like being in a shell where cwd is '/playbooks/'.  :)20:41
mordredjeblair: and if geerlingguy wanted to do his role but keep it that way, he could always add roles: role_under_test: https://github.com/geerlingguy/ansible-role-java to his job20:41
tobiashjeblair: yes, that was the thought20:42
mordredI'm _less_ crazy about the leading / - because that, to me, implies the root of the filesystem, but we're more likely to be in /home/zuul (or similar) - but I don't feel SUPER strongly about it20:42
jeblairmordred: i'm trying to catch up -- where's geerlingguy's role_under_test ?20:42
mordredhttps://github.com/geerlingguy/ansible-role-java/blob/master/tests/test.yml#L1020:43
jeblairmordred: yeah, but where's that role?20:43
jlkmaking executor do "smart things" with role names and paths makes it harder for people to replicate outside of zuul20:43
mordredhe has a wrapper script he wget's into his travis job that puts his role in place and always names it "role_under_test"20:43
jeblairmordred: gotcha.  yes.  i think that's a substandard experience that we can support in the way you described with the roles: mapping.  :)20:44
mordredjeblair: https://gist.githubusercontent.com/geerlingguy/73ef1e5ee45d869457020:44
tobiashmordred: so you're in favor of enforcing (repo) absolute paths for pre/post playbooks and default to playbooks/<jobname> for main playbook?20:46
mordredjeblair, jlk: here's another post on dealing with a similar issue- which involves putting a symlink in a roles dir in the tests dir: https://www.ansible.com/blog/testing-ansible-roles-with-docker20:47
mordredscroll down to "A Role Under Test"20:47
jeblairmordred: wow, that symlinks outside of the repo20:50
jeblair(it doesn't have to though, so we can probably just ignore that.  just it's an interesting example)20:50
mordredtobiash: yes - I think that's nicer personally- the least amount of magic (the only magic that happens is if you're getting magical implied-run-name magic too)20:53
tobiashmordred: wfm20:53
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: wip: support custom playbook locations  https://review.openstack.org/47767220:55
tobiashmordred: like that? ^^20:55
tobiashtests would have to be fixed (and commit message beautified)20:55
jlkjeblair: reading through doc update, does an executor (if ran without a stand alone merger) not need any of the config entries from the [merger] section of the configuration?20:58
tobiasheod for me now, cya20:58
*** dkranz has quit IRC21:01
jeblairjlk: it uses git_user_name and git_user_email from the merger section21:04
jeblairthat's another thing missing from my current docs stack.  :)21:04
jlkyeah, I have a pending comment21:04
jeblairi wonder if we should continue doing that, or duplicate the options for executor?21:05
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Implement pipeline requirement of github labels  https://review.openstack.org/47396221:15
openstackgerritMerged openstack-infra/zuul feature/zuulv3: config: refactor config get default  https://review.openstack.org/47448421:20
openstackgerritMerged openstack-infra/zuul feature/zuulv3: bubblewrap: adds --die-with-parent option  https://review.openstack.org/47316421:25
openstackgerritMerged openstack-infra/zuul feature/zuulv3: executor: run trusted playbook in a bubblewrap  https://review.openstack.org/47446021:25
dmsimardI'm trying to know where to look (nodepool, zuul, or the disk image), before Zuul 2.5, how did the jenkins slave.jar end up on the nodepool VM and executed so the node connected back ot jenkins ?21:32
dmsimards/ot/to/21:32
jlkwouldn't that be a disk image builder element?21:32
dmsimardjlk: I've been grepping 'jar' left and right and nothing's coming up21:34
dmsimardI'm probably not looking in the right places21:34
mordreddmsimard: we had puppet21:35
jlkyeah I would think it's the infra configuration21:35
jlkfor dib21:35
mordreddmsimard: that we ran that installed java on the slave21:35
mordreddmsimard: but jenkins itself is responsible for copying the jar as part of teh ssh slave plugin21:35
mordreddmsimard: and jenkins connected to the slave, not the other way around21:35
dmsimardmordred: ah, that's a good hint, thanks21:35
mordreddmsimard: the attach was a REST API call that added the node as an ssh slave - which caused jenkins to connect to the slave21:36
jlkoooh, somebody should go through the github driver code and wire up some statsd metrics.21:36
dmsimardmordred: ok, that's exactly it -- java doesn't end up being installed in the upstream images anymore. Thanks <321:38
mordreddmsimard: \o/21:38
mordredclarkb, fungi: ^^ that may be a thing we want to take note of WRT any 3rd party ci using zuul v2 dib and puppet21:39
mordredand still using jenkins21:39
clarkboh I ran into that with gerrit build jobs and didn't make the connection to "this will break jenkins"21:47
jeblairzuul meeting time in #openstack-meeting-alt22:00
jamielennox'22:01
*** hashar has quit IRC22:41
mordredjlk: looks like the PR 'body' isn't even in the comments in https://api.github.com/repos/j2sol/z8s-sandbox/issues/2/comments22:43
mordredjlk: only in the body on https://api.github.com/repos/j2sol/z8s-sandbox/pulls/222:44
jlkhaha, wtf.22:45
jlkso, that's actually maybe better?22:45
jlkOOOH22:46
jlkso, one flaw in my investigation, when looking at the webhook event that github sends, it doesn't show what TYPE of event. I had assumed it was a comment type event, but in reality it may have been a "pull_request" event, and the action was "edited"22:47
jlkso... that would make more sense. The pull_request was edited, the body was updated.22:47
jlkthis is workable22:47
jlkhttps://developer.github.com/v3/activity/events/types/#pullrequestevent   derp.22:48
rbergeronjlk: i uh.... just informed this "maxamillion" character about bonny23:37
rbergeronso um23:37
rbergeronif you hear from him... that's why :D23:38
rbergeronalso: jlk -- I have kind of read / experimented with the github api in like, gross ickiness types of detail -- not that i expect to know more than you, but perhaps I can lend the "dumb person who knows enough to be dangerous" aspect while you are playing with it -- between screwing with it for data and also in ELK scenarios and stupid github bigquery23:40
rbergeroni like to think i know... at least something that could be slightly potentially useful momentarily23:40
jlkawesomes!  Thanks for the heads up23:41
jlkWe are using some searching stuff, and thought we'd need more, but maybe not.23:41
jlkoh, like, I do need to be able to search github for any PRs with "Depends-On" and "org/project/pull/number" as an initial comment.23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!