| openstackgerrit | Ian Wienand proposed openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem https://review.openstack.org/511079 | 00:02 |
|---|---|---|
| *** xinliang has quit IRC | 00:23 | |
| *** xinliang has joined #zuul | 00:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 00:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 00:43 |
| *** jkilpatr has quit IRC | 01:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 01:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 01:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 01:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 01:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 01:38 |
| *** pleia2 has quit IRC | 02:43 | |
| *** maxamillion has quit IRC | 02:43 | |
| *** maxamillion has joined #zuul | 02:45 | |
| *** pleia2 has joined #zuul | 02:50 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add upload-npm role https://review.openstack.org/510686 | 04:08 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: zuul-cloner-shim: Use st_dev to check for filesystem https://review.openstack.org/511079 | 04:08 |
| *** openstackgerrit has quit IRC | 07:03 | |
| *** hashar has joined #zuul | 07:24 | |
| kklimonda | should zuul load new configuration from its config-projects when running new jobs, or is it expected that changes made to repos defined in config-projects are not picked until we reload zuul configuration | 08:37 |
| *** electrofelix has joined #zuul | 08:45 | |
| *** hashar is now known as hasharAway | 09:50 | |
| *** pbrobinson has quit IRC | 10:29 | |
| *** pbrobinson has joined #zuul | 10:31 | |
| tobiash | kklimonda: config repo changes are picked up on merge | 10:43 |
| kklimonda | tobiash: that assumes that config repo is being driven through zuul? | 10:44 |
| tobiash | kklimonda: you just need to get the merge event | 10:48 |
| *** jkilpatr has joined #zuul | 10:49 | |
| kklimonda | and another question - for config projects, master branch is hardcoded (I assume for security reasons) but that makes it hard to have two environment (one for production zuul and one for staging zuul). Is there anything preventing us from expanding tenant config to point which branch should zuul pull config from? | 10:52 |
| *** mrhillsman has quit IRC | 11:03 | |
| *** mattclay has quit IRC | 11:03 | |
| *** mrhillsman has joined #zuul | 11:05 | |
| *** mattclay has joined #zuul | 11:06 | |
| *** dkranz has joined #zuul | 12:38 | |
| *** hasharAway is now known as hashar | 13:01 | |
| mordred | kklimonda: I do not know the answer to that question - I believe we'll need to talk to jeblair about that | 13:15 |
| mordred | kklimonda: I *think* it should be possible - but it's not a scenario that has come up before, at leats not in my brainhole | 13:15 |
| *** hashar is now known as hasharAway | 13:55 | |
| jeblair | kklimonda, mordred: i think we can expand the tenant config to specify the branch (or tag, or other ref) used for config-projects. | 14:03 |
| mordred | jeblair: cool. that was my first thought - but then I worried about implied branchmatchers- then I realized that's not an issue for config projects - then I confused myself :) | 14:16 |
| jeblair | mordred: ya, i think that's all correct. including the confusing yourself bit. very important. | 14:17 |
| *** openstackgerrit has joined #zuul | 14:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror https://review.openstack.org/511253 | 14:17 |
| mordred | jeblair: it's theMOST important | 14:18 |
| *** bhavik1 has joined #zuul | 16:09 | |
| *** weshay is now known as weshay|ruck | 16:32 | |
| *** bhavik1 has quit IRC | 16:43 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Also add security.list for Debian on configure-mirror https://review.openstack.org/511253 | 16:54 |
| dmsimard | is v3 publishing to mqtt with https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/puppetmaster/mqtt.py ? | 17:42 |
| mordred | dmsimard: pabelanger started work on an mqtt reporter plugin I believe, but we've got it on hold until post rollout | 17:43 |
| dmsimard | mordred: ok so we wouldn't be using the callback then ? | 17:43 |
| pabelanger | yup, pending rollout I was going to get back into it | 17:44 |
| mordred | dmsimard: no - we are not | 17:44 |
| dmsimard | mordred: ok. | 17:44 |
| dmsimard | FWIW TripleO is trying to do something with ara and graphite https://review.openstack.org/#/c/479882/7/roles/collect-logs/library/ara_graphite.py | 17:46 |
| jeblair | dmsimard: neat, though i don't think that's going to work since we firewall the graphite server to specific hosts | 17:47 |
| dmsimard | jeblair: tripleo has their own graphite server :( | 17:47 |
| pabelanger | Ya, I am not impressed that is a thing | 17:48 |
| dmsimard | +1 | 17:48 |
| jeblair | dmsimard: this is maybe a conversation for #openstack-infra | 17:48 |
| dmsimard | sure, I was rabbit holed' into graphite/stats/callback things | 17:49 |
| *** electrofelix has quit IRC | 17:55 | |
| mordred | dmsimard, jeblair: on a day that is not today, I think there's 5 potentially interesting things we could explore or discuss related to that conversation: a) mqtt zuul reporter b) statsd/graphite stats as ansible callback in the ansible run by zuul c) mqtt events as ansible callbacks in the ansile run by zuul d) how to best incorporate such things (mqtt, statsd, ARA) from ansible run on remote nodes as part of | 18:06 |
| mordred | job content e) how best to incorporate things like statd counters from non-ansible remote node content (like the SpamapS spec) | 18:06 |
| dmsimard | right, and there's some overlap too. | 18:08 |
| jeblair | i agree those are among the things to discuss. :) | 18:08 |
| dmsimard | I've had people ask me to make ara export things to graphite, influx and elasticsearch and so and so, while there are perhaps "native" ansible callbacks to do that | 18:08 |
| mordred | yes indeed - as well as some pieces that I legitimately do not know the right answer to - d and e come to mind | 18:08 |
| SpamapS | dmsimard: just a personal opinion here: If you could make ARA more interactive.. I'd support you ignoring everything else that everyone asks you to do with it. ;) | 18:09 |
| dmsimard | SpamapS: interactive? | 18:10 |
| SpamapS | dmsimard: yes, my people would like to watch runs as they happen. | 18:10 |
| dmsimard | SpamapS: I suck at frontend but happy to review contributions :D | 18:11 |
| dmsimard | SpamapS: the data is there (updated in the database as the playbook progresses), the fact that it's not updated in the realtime in the UI is a frontend thing | 18:11 |
| jeblair | dmsimard: you know you're not fooling anyone. no one believes you when you say you suck at frontend. or anything really. :) | 18:12 |
| dmsimard | jeblair: if only you knew how much copy pasta was involved in the making of the UI :D thanks god for stackoverflow and code snippets from http://www.patternfly.org/pattern-library/ | 18:12 |
| pabelanger | Yah, you suck much less at frontend them me :) | 18:13 |
| dmsimard | I mean, mnaser is a pro, he went like "no you can't nest a link inside a button" like yeah, that's super obvious :p | 18:13 |
| mnaser | hi | 18:13 |
| * mnaser reads buffer | 18:13 | |
| dmsimard | mnaser: nothing important lol | 18:13 |
| mnaser | oooh | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 18:14 |
| mnaser | that would be cool | 18:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 18:14 |
| mnaser | a small instance of ara that runs on ever worker node | 18:14 |
| mnaser | which you can enter to see what it's doing | 18:14 |
| *** hasharAway is now known as hashar | 18:14 | |
| dmsimard | jeblair: on the other hand, you rock at doing console graphics with presentty and gertty :p | 18:15 |
| dmsimard | mnaser: yeah, I suspect we'll want some other way of presenting ara reports. static reports are nice but not very efficient. | 18:16 |
| SpamapS | my dream Ansible experience is that currently running playbooks are shown with the current task, and if clicked, show the streaming output of any commands. | 18:21 |
| mordred | SpamapS: I think we can make the plumbing of a lot of that happen | 18:22 |
| SpamapS | mordred: I think we kind of already have. :) | 18:22 |
| SpamapS | just needs tying together | 18:22 |
| mordred | SpamapS: well -I meant in an upstream-ansible-enabled kind of way | 18:22 |
| SpamapS | mm yes | 18:22 |
| mordred | SpamapS: thething we're doing for streaming output is ... well, there are some other things I intend to explore as soon as I've got time to explore them | 18:22 |
| mnaser | sounds so web 2.0 | 18:22 |
| mnaser | :-P | 18:22 |
| mordred | mnaser: I'm still trying to catch up to web 0.5 | 18:23 |
| mnaser | :P | 18:23 |
| * SpamapS is webscale | 18:23 | |
| dmsimard | SpamapS: tower does that I think | 18:23 |
| dmsimard | SpamapS: but it's also able to do that by doing a lot of invasive things | 18:24 |
| dmsimard | (that ara doesn't do, ofc, ara is super transparent) | 18:24 |
| mordred | dmsimard: except without the output streaming - the tower folks are also interested in that being a thing in ansible sothey can use it too | 18:24 |
| dmsimard | mordred: s/output/task output/ | 18:24 |
| dmsimard | mordred: you can stream the playbook stdout, but not the task output | 18:25 |
| dmsimard | at least iirc | 18:25 |
| dmsimard | well, not any different than what you'd see if you ran ansible in your terminal, I mean | 18:25 |
| mordred | yes - just like the command line- it's stremaing task output that makes it even more exciting | 18:25 |
| mordred | speaking of - the tower architecture for executing ansible playbooks is REMARKABLY similar to ours - they git clone stuff into work dirs then run the ansible in a bubblewrap :) | 18:26 |
| dmsimard | oh yeah ? they use bubblewrap too ? huh. | 18:26 |
| dmsimard | I didn't dig at the code too much, only saw their ara callback equivalent using memcached and I slowly turned away | 18:27 |
| pabelanger | Ya, but last I heard, they want to stop using it | 18:27 |
| pabelanger | atleast talking to tower folks (matt?) at ansiblefest SF | 18:27 |
| mordred | pabelanger: they want to stop using bubblewrap? | 18:41 |
| pabelanger | mordred: ya, they were talking about moving to another container, trying to remember what it was | 18:42 |
| mordred | pabelanger: I'm guessing OCI | 18:43 |
| Shrews | more importantly, what are their reasons for wanting to stop? maybe they know things we do not | 18:43 |
| pabelanger | ya, I asked if it was because of something specific in bwrap, but IIRC, the response was just wanting something more embrased in the community. | 18:44 |
| pabelanger | not many things are using bwrap | 18:44 |
| pabelanger | mordred: could have been OCI, but not certain | 18:44 |
| pabelanger | something, something, openshift? | 18:45 |
| pabelanger | _shurgs_ | 18:45 |
| dmsimard | Shrews: fyi 2.4.1rc1 should land today and release wednesday next week if there are no slips | 18:46 |
| SpamapS | mordred: haha bubblewrap too, that's awesome. ;) | 18:48 |
| SpamapS | It's probably so they can just point at a k8s and not worry about executor load. | 18:48 |
| dmsimard | SpamapS: so you'll have to run k8s if you want to run tower ? :/ | 18:50 |
| dmsimard | hopefully it's an optional backend, I suspect that might turn some people off | 18:50 |
| mordred | dmsimard: woot! | 18:52 |
| Shrews | dmsimard: yup. will have to get mine and mordred's changes for 2.4 in after cutover | 18:58 |
| mordred | Shrews: wait - we have changes for 2.4 still pending? | 18:59 |
| mordred | Shrews: OH | 18:59 |
| mordred | thezuul changes | 18:59 |
| mordred | *duh* | 18:59 |
| Shrews | mordred: https://review.openstack.org/505354 | 18:59 |
| Shrews | i think dmsimard left a question for you there | 18:59 |
| Shrews | yeah, zuul | 19:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 19:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 19:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 19:02 |
| kklimonda | so, a crazy thought - should I expect job variants to work with job inheritance, where child jobs will pick a matching parent variant? something similar to https://gist.github.com/kklimonda/0de97f5bbb9551f86f75223caf219880 | 19:14 |
| dmsimard | kklimonda: you can't redefine a job more than once | 19:15 |
| dmsimard | kklimonda: unless I'm missing something, there's 3 'job-base' jobs defined | 19:15 |
| kklimonda | dmsimard: those are job variants - basically job overloading? ;) You can define jobs that share name, but have different criteria for selection, and zuul will pick the right one. But I think I'm expecting too much by asking zuul to pick the correct parent job variant, and I'm wondering if that's a bug, an expected behavior, or a limitation. | 19:19 |
| mordred | kklimonda: I *think* that should work ... | 19:20 |
| mordred | jeblair: ^^ | 19:20 |
| dmsimard | kklimonda: hmm, interesting, I didn't expect that to work but I see where you're going | 19:21 |
| jeblair | kklimonda: yeah, line 17 is only going to inherit from line 2 | 19:22 |
| kklimonda | indeed, that's what is happening | 19:23 |
| mordred | ah - I was wrong. cool. I learned something today | 19:23 |
| jeblair | it's... conceivable we could change that. we'd have to make inheritance later binding (like variance). but for the moment at least, this is the intention. | 19:23 |
| jeblair | it doesn't look like that made it into the documentation though :) | 19:24 |
| Shrews | we do have a discussion on variants and matching in https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html?highlight=variant#job | 19:25 |
| jeblair | yeah, and we even mention the 'reference definition' but don't really indicate what's so special about it. :) | 19:26 |
| jeblair | i wonder if we should reverse the order of discussion of variance and inheritance, then mention the reference definition in the inheritance pgraph | 19:27 |
| dmsimard | It's not the first way I'd think about defining job variants | 19:28 |
| dmsimard | As in, it's probably not intuitive to write things that way. My reflex would be to do like we do now, have a base job and then define "variants" as different jobs inheriting from the parent job. I guess the use case is not super obvious to me. | 19:29 |
| mordred | dmsimard: I could see having a thought like "the stable/newton branch of the devstack base job should run on ubuntu-trusty while the master branch should run on ubuntu-xenail" | 19:34 |
| mordred | and having people who consume the devstack base job expect that information to make it into their child jobs | 19:34 |
| dmsimard | I might be stuck in the concept of 'the name of the job must describe what it's doing and what it's running on' | 19:35 |
| dmsimard | What we're talking about here is making what is ultimately a single job do different things | 19:36 |
| dmsimard | ¯\_(ツ)_/¯ | 19:36 |
| kklimonda | dmsimard: with jobs variants my project has only one job "package-build-xenial", but with inheritance I'd probably need separate jobs "package-build-xenial-R4.0" and "package-build-xenial-master" with branch filters. | 19:36 |
| dmsimard | kklimonda: but you end up having to define those anyway right ? the only thing you might be saving on is the project layout definition ? | 19:37 |
| kklimonda | dmsimard: indeed, although that save can be substantial - If I ever finish the migration we're talking about defining 5 jobs per project vs 25 jobs per project. | 19:40 |
| dmsimard | kklimonda: I have to step away momentarily but you got me curious and I must dig deeper to better understand the use case :) | 19:41 |
| *** harlowja has quit IRC | 20:00 | |
| jeblair | mordred: yeah that's a pretty compelling case for late-binding inheritance | 20:13 |
| jeblair | dmsimard: i don't think the name of the job should describe what it's running on. it should just describe what the job does. how it does it may be different on different branches. recognizing that is pretty fundamental to zuulv3 and it's why we have variants. | 20:16 |
| dmsimard | jeblair: It's probably down to personal preference, but I understand how people could prefer otherwise. For me the cost of having a more explicit job name outweighs the cost of having to go *inside* the job to figure out what actually ran | 20:24 |
| dmsimard | kklimonda: I'm back and I ran with your gist because I want to make sure I understand the use case, unless mistaken, do we agree that all 3 files in https://gist.github.com/dmsimard/b5a85109cffde40c5e6699bdc4e5c2d1 are equivalent ? (Your example is the first one) | 20:25 |
| dmsimard | wait no, silly gist ordering, your example is 'original.yaml' | 20:28 |
| jeblair | dmsimard: there's a certain amount of personal preference involved, but there's also very clearly a use case that zuulv3 was designed to support. and that is: "the single job tox-py27 should use a trusty node on this branch, and a xenial node on this other branch". | 20:30 |
| jeblair | so i want to be very clear that is absolutely a first-class thing, and we should discourage no one from doing that, or suggest it's any kind of advanced feature. | 20:30 |
| jeblair | it's the third example in the zuulv3 spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#jobs | 20:31 |
| jeblair | (though that calls them "aspects" rather than "variants" -- things evolved slightly in implementation) | 20:32 |
| dmsimard | jeblair: Yeah, I didn't realize this was a thing until earlier, thanks. I haven't yet encountered that use case but if I did (before knowing this), my first reflex as an end user would've been to do separate jobs based off of a parent like this: https://gist.githubusercontent.com/dmsimard/b5a85109cffde40c5e6699bdc4e5c2d1/raw/d790b3c807fe322933db0126acb205716c734b9c/no-templates.yaml | 20:35 |
| jeblair | dmsimard: well, hopefully we'll improve the docs to the point where people know how to use variants. it's a tool to simplify the config and reduce repitition, so that example can be written as http://paste.openstack.org/show/623381/ | 20:41 |
| dmsimard | jeblair: right, it's pretty powerful. | 20:42 |
| *** dkranz has quit IRC | 20:44 | |
| jeblair | so i think the actionable feedback here is that we may have users that expect that variants apply to items in the inheritance chain. we should decide whether that's something that's safe to do, whether it's more or less confusing than the current state, and then either implement support for it, or document the status quo better. | 20:46 |
| jeblair | i'm not in a good position to consider that carefully right now, and it's not a trivial change to config loading. so i will, at least, personally be sleeping on that one. :) | 20:47 |
| *** jkilpatr has quit IRC | 20:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Switch statsd config to zuul.conf https://review.openstack.org/511017 | 21:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some stats from executor https://review.openstack.org/511073 | 21:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Emit some nodepool stats https://review.openstack.org/511085 | 21:06 |
| SpamapS | just a note, it might be an excellent teaching tool to include exactly those two things in an in-depth "getting started" guide. | 21:06 |
| SpamapS | "You might think you would have to ...{dmsimard's thing}. But it's simpler! You just need to {jeblair's thing}" | 21:07 |
| mordred | SpamapS: "dmsimard wrote two jobs ... you'll never guess what happened next!" | 21:07 |
| dmsimard | simpler is relative (or subjective), shorter would be an appropriate term :p | 21:07 |
| *** harlowja has joined #zuul | 21:07 | |
| kklimonda | @dmsimard yes, all 3 are equivalent and I agree it's just a matter of preference - although jebrail's example show how job variants have a slight edge (but again, it's probably a matter of perspective ;)) | 21:08 |
| dmsimard | SpamapS, mordred: but yes, we could totally take one set of jobs and write them in different ways which makes them equivalent a bit like I did here https://gist.github.com/dmsimard/b5a85109cffde40c5e6699bdc4e5c2d1 | 21:09 |
| dmsimard | s/write them/document them/ | 21:09 |
| dmsimard | although it's already well explained in the docs if you pay attention :) | 21:10 |
| kklimonda | btw, will zuul ignore jobs that don't match the branch completely, and not report back "SKIPPED"? | 21:10 |
| dmsimard | kklimonda: correct | 21:10 |
| jeblair | yeah, skipped is only for job dependencies not met, or similar. | 21:11 |
| jeblair | like, the upload job was skipped because the build job failed | 21:11 |
| *** jkilpatr has joined #zuul | 21:27 | |
| *** hashar has quit IRC | 21:37 | |
| SpamapS | mordred: I so want to see all new user guides written as clickbait | 21:55 |
| clarkb | You'll never believe what monty did to your zuul configs! | 21:55 |
| *** jkilpatr has quit IRC | 21:57 | |
| *** jkilpatr has joined #zuul | 21:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 23:28 |
| jeblair | kklimonda, dmsimard, mordred: ^ that's not a decision; that's part of my thought process. | 23:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: experiment with late-binding inheritance https://review.openstack.org/511352 | 23:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use weakref for change cache https://review.openstack.org/511355 | 23:45 |
| jeblair | jlk: ^ i haven't tested that yet, but think it may be of interest | 23:45 |
| jlk | ohai | 23:45 |
| jlk | "holf references" lol :D | 23:46 |
| jeblair | jlk: wow. i don't know what that means, but it really does sound exactly like what it does. :) | 23:47 |
| jlk | that looks like it would work | 23:49 |
| jlk | nothing in github seems to delete an item from the cache though. Maybe that's okay? | 23:50 |
| jlk | wait, no | 23:51 |
| jlk | We do the deletion as well in github, but apparently done in a way that was more compatible with the weakref than gerrit was? | 23:51 |
| jeblair | jlk: i think the only explicit deletions right now are on errors (ie, don't put a broken change object in the cache) | 23:52 |
| jeblair | jlk: and yeah, we had a two-layer cache in the gerrit driver, so i changed it to use the one-layer (with a tuple as key) like the github driver | 23:52 |
| jlk | gotcha | 23:53 |
| jeblair | we might actually be able to promote some of this up an api level; i haven't thought that all the way through yet | 23:53 |
| jeblair | though it is now so simple i'm not sure that'd get us much :) | 23:53 |
| jlk | heh | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!