| *** saneax is now known as saneax-_-|AFK | 00:22 | |
| *** jamielennox is now known as jamielennox|away | 04:13 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove extra debugging around git repo leaks https://review.openstack.org/443317 | 04:33 |
|---|---|---|
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change mutex to counting semaphore https://review.openstack.org/442563 | 04:43 |
| *** jamielennox|away is now known as jamielennox | 04:47 | |
| *** isaacb has joined #zuul | 05:00 | |
| *** isaacb has quit IRC | 05:16 | |
| *** isaacb has joined #zuul | 05:29 | |
| *** isaacb has quit IRC | 05:33 | |
| *** yolanda has quit IRC | 05:41 | |
| *** saneax-_-|AFK is now known as saneax | 06:03 | |
| *** isaacb has joined #zuul | 06:08 | |
| *** bhavik1 has joined #zuul | 06:11 | |
| *** isaacb has quit IRC | 06:12 | |
| *** bhavik1 has quit IRC | 06:21 | |
| *** isaacb has joined #zuul | 06:24 | |
| *** isaacb has quit IRC | 06:30 | |
| *** isaacb has joined #zuul | 06:30 | |
| *** isaacb has quit IRC | 06:33 | |
| *** isaacb has joined #zuul | 06:34 | |
| *** abregman has joined #zuul | 06:45 | |
| *** jamielennox is now known as jamielennox|away | 07:09 | |
| *** jamielennox|away is now known as jamielennox | 07:27 | |
| *** isaacb has quit IRC | 07:43 | |
| *** isaacb has joined #zuul | 07:43 | |
| *** isaacb has quit IRC | 07:53 | |
| *** hashar has joined #zuul | 08:03 | |
| *** isaacb has joined #zuul | 08:15 | |
| *** abregman has quit IRC | 08:38 | |
| *** abregman has joined #zuul | 08:51 | |
| *** isaacb has quit IRC | 09:12 | |
| *** isaacb has joined #zuul | 09:45 | |
| *** hashar has quit IRC | 09:59 | |
| *** hashar has joined #zuul | 10:08 | |
| *** isaacb has quit IRC | 11:07 | |
| Shrews | morning zuulers | 11:09 |
| *** abregman has quit IRC | 11:09 | |
| *** abregman has joined #zuul | 11:24 | |
| *** isaacb has joined #zuul | 11:40 | |
| tobiash_ | hi, I've a question about config loading in zuulv3 | 12:24 |
| tobiash_ | is it correct that all config and project repos must be mentioned in the tenant config file? | 12:25 |
| tobiash_ | or is it possible to extend the list of config/project repos in a config repo? | 12:25 |
| tobiash_ | that would be cool as it would make it possible to just deploy a minimal tenant config on the server and let the config loader load all other config/project repos directly from git | 12:30 |
| *** isaacb has quit IRC | 12:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Add back statsd reporting https://review.openstack.org/443605 | 12:33 |
| *** isaacb has joined #zuul | 13:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Add back statsd reporting https://review.openstack.org/443605 | 13:04 |
| Shrews | pabelanger: jeblair: Not being familiar with statsd, I'm going to need some input on how to deal with updating stats in the new world. Recording launch stats is still straightforward, but updating node stats is where I'm not sure how statsd will react to what I have in 443605. | 13:07 |
| Shrews | pabelanger: jeblair: In the new world, we can't assume our config has all known providers or node types, so we can't record data for things we don't know about, like the old updateStats() call could. | 13:08 |
| Shrews | Right now I'm guessing at how to record the stats properly, and don't know how statsd will react to a value not being record at all. | 13:10 |
| Shrews | not being recorded* | 13:12 |
| mordred | Shrews: it's generally pretty good about not receiving things - what sort of things are you worried about not being recorded? | 13:20 |
| mordred | Shrews: oh - so also, for the things in updateStats (I see where you're worried about one thing not knowing everything) | 13:22 |
| mordred | Shrews: you can just inject an entry for the launcher ... so like, instead of #nodepool.provider.PROVIDER.nodes.STATE ... you could do #nodepool.launcher.LAUNCHER.provider.PROVIDER.nodes.STATE | 13:23 |
| mordred | and then on the graphite site, we can make graphs of nodepool.launcher.*.provider.foo.nodes | 13:24 |
| Shrews | mordred: well, we've already decided that sharing a provider across launchers isn't something we plan to do. but you're right in that my worry is i can't report about providers that other launchers may know about. | 13:35 |
| Shrews | mordred: i guess as long as we can get it reporting "something" for now, we can always tweak it later | 13:36 |
| Shrews | maybe adding launcher id is something we should do | 13:37 |
| Shrews | mordred: also, i'm ever so happy that my brain woke me up at 6am, demanding to get this code out of my head :) | 13:39 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add back statsd reporting https://review.openstack.org/443605 | 13:50 |
| Shrews | we can tweak what's there if we have to | 13:50 |
| mordred | Shrews: I love 6am wakeup | 14:01 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove old/dead classes https://review.openstack.org/443644 | 14:01 |
| *** isaacb has quit IRC | 14:14 | |
| *** isaacb has joined #zuul | 14:20 | |
| *** saneax is now known as saneax-_-|AFK | 14:35 | |
| *** abregman is now known as abregman|mtg | 15:00 | |
| jeblair | tobiash_: there is not currently a way to extend the tenant configuration from a project or config repo | 15:05 |
| tobiash_ | jeblair: would that be a desired feature? | 15:06 |
| tobiash_ | I'm currently in progress of setting up a zuulv3 instance for playing around with it as early as possible | 15:06 |
| jeblair | tobiash_: i'm not sure... it's dangerous, so it will need some extra protection (i think the tenant config file needs to be able to say what kinds of things each project/config repo is allowed to load; if we add that, we will want to add something that lets us specify whether a repo is permitted to load tenants). it would also be a moderately large change to the configloader, as it is loaded fairly early in the process. | 15:10 |
| tobiash_ | jeblair: ok, then I will put the tenant config into a git repo, update it outside of zuul and trigger a reload | 15:13 |
| tobiash_ | like we do it currently also with the layout.yaml in current zuul | 15:14 |
| tobiash_ | jeblair: I have another question. Is the zuul-merger still required or does the zuul-launcher completely replace the zuul-merger functionality? | 15:21 |
| jeblair | tobiash_: the launcher contains all the functionality of a merger, so you should not need to run one. we're keeping it around for the moment because we think there might be enough work for a merger to do that it may be useful to scale it out on its own. | 15:23 |
| tobiash_ | ah, thx | 15:23 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add leaked instance cleanup https://review.openstack.org/443690 | 15:24 |
| pabelanger | jeblair: Shrews: mind +3 on 443306? our gitpython cap for master | 15:26 |
| *** Cibo_ has joined #zuul | 15:37 | |
| jeblair | Shrews: tiny bikeshed nit on 443605, but otherwise, i think that'll be fine. | 15:41 |
| jeblair | Shrews: (the thing i'm most worried about is resetting gauges to zero. if we traumatically lose a provider while it has nodes outstanding, and never recover it, we won't ever send a zero gauge value. i can't think of a way to reconcile that at the moment, except by putting all the provider names in zookeeper (like how we register launchers). maybe that's what we should end up doing, but that's probably a followup change for later) | 15:43 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: WIP: Remove dead code from NodePool class https://review.openstack.org/443714 | 15:44 |
| Shrews | jeblair: yeah, that was one of my worries, especially not being familiar with how statsd operates | 15:45 |
| Shrews | but we can improve on what's there | 15:45 |
| jeblair | Shrews: there's an option to have it automatically delete idle gauges, but i'm pretty sure idle means "hasn't updated in 10 seconds" which i don't think will work for us here. | 15:46 |
| mordred | jeblair: yah - I don't think 10 seconds is idle enough for us | 15:52 |
| mordred | jeblair: maybe as a workaround for your traumatic case, we can just provide a simple utility script like "zero_provider_stats" or something - because that's an extreme enough case that just having someone choose to blank out something seems non-terrible | 15:53 |
| *** isaacb has quit IRC | 15:54 | |
| jeblair | mordred: i would have thought so a while ago, but it's happened enough to us that i think we'll want to make it automatic | 15:58 |
| mordred | jeblair: okie | 15:59 |
| *** abregman|mtg is now known as abregman | 16:06 | |
| *** isaacb has joined #zuul | 16:19 | |
| openstackgerrit | Merged openstack-infra/zuul master: Cap GitPython at 2.1.1 due to performance degradation https://review.openstack.org/443306 | 16:28 |
| *** Cibo_ has quit IRC | 16:50 | |
| tflink | are there any public logs of zuul v3 runs and/or a test I could look at? | 16:54 |
| jeblair | tflink: yes, it's starting to run tests on itself, so you can look at any recent change to the zuulv3 repo... i will find you some interesting ones | 16:55 |
| tflink | thanks, I wasn't sure where to look | 16:55 |
| jeblair | tflink: these are good: https://review.openstack.org/#/q/topic:zuulv3-ansible | 16:56 |
| jeblair | tflink: in those, pabelanger is using zuulv3's dynamic in-tree configuration to create some initial sample jobs | 16:56 |
| jeblair | tflink: (we're in the middle of trying to decide which of the approaches in the first 2 patches to go with) | 16:57 |
| jeblair | tflink: if you follow the link to one of those changes, you'll see a box with "Zuul (1 rechecks)" in, i dunno, sort of the middle-right-hand-top of the page? :) | 16:57 |
| jeblair | tflink: links under there go to zuulv3 runs. under that are the "Jenkins" runs from zuulv2 | 16:58 |
| *** bhavik1 has joined #zuul | 16:59 | |
| tflink | jeblair: to make sure I'm understanding, those tests that v3 is running on itself are being generated at runtime right now? | 16:59 |
| tflink | I'm looking at stealing how the test playbooks are laid out for some fedora stuff as some background | 17:00 |
| * tflink thought that format/layout had been mostly figured out but may have been thinking about the generated playbook from v2.5 | 17:00 | |
| jeblair | tflink: yes, they are running on the change that proposed running them :) | 17:01 |
| jeblair | tflink: it's *mostly* figured out, but still somewhat in flux. | 17:01 |
| jeblair | tflink: we still have some known changes to the config syntax coming. mostly what we're doing in those changes is experimenting with various approaches to building out our system of jobs (we have a lot of jobs, so it warrants a few minutes thought as to what the basic building blocks look like) | 17:02 |
| tflink | jeblair: not sure I follow what config syntax you're talking about. is that something used by ansible or more in how zuul itself is configured | 17:04 |
| jeblair | tflink: the zuul.yaml syntax | 17:05 |
| Shrews | jeblair: your second comment on https://review.openstack.org/#/c/443690/1/nodepool/nodepool.py ... are you suggesting we store the provider name in the instance metadata? Otherwise I'm not sure what values you're suggesting we compare | 17:05 |
| jeblair | Shrews: yes, in nodepool master we store provider_name in instance metadata; if we missed that in v3 i think we should add it. | 17:06 |
| Shrews | jeblair: yeah, i don't see that in the createServer() | 17:06 |
| *** bhavik1 has quit IRC | 17:06 | |
| Shrews | oh! nm, i see it | 17:07 |
| Shrews | it was cleverly hidden :) | 17:07 |
| mordred | jeblair: I don't remember - did we ever land the change to move away from json-encoding the things in the metadata? | 17:07 |
| tflink | jeblair: ok, thanks. I don't think this current thing can wait for all the dust to settle but I wanted to make sure that we didn't go off in a completely different direction :) | 17:07 |
| jeblair | mordred: no: 297950 | 17:08 |
| jeblair | mordred: it has a -1 nit from me and a merge conflict | 17:08 |
| mordred | oh! I now see that | 17:08 |
| *** abregman is now known as abregman|mtg | 17:10 | |
| jeblair | tflink: cool. btw rbergeron will be sending out periodic (biweekly or more as needed) status update emails; should be a low-impact way for you to be informed of milestones. | 17:10 |
| * tflink wonders if he remembered to sign up for that mailing list | 17:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 17:12 |
| pabelanger | regarding zuulv3-ansible playbooks, we've had consistent voting in favor of 438281 (multi-playbooks) today. Mind you, nobody has said anything negative about 442180 either, just easier to process multi playbooks | 17:13 |
| *** isaacb has quit IRC | 17:14 | |
| clarkb | ya for me its really just nice to have that high level summary view without a ton of details mixing in | 17:15 |
| clarkb | kind of like what projects.yaml once was | 17:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove old/dead classes https://review.openstack.org/443644 | 17:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add leaked instance cleanup https://review.openstack.org/443690 | 17:19 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add back statsd reporting https://review.openstack.org/443605 | 17:19 |
| jeblair | i suspect if we get a lot of roles being copied around to all the playbooks (bindep, install db, check sudo, check oom, subunit2html, etc), we might want to refactor a bit later. we have a lot of choices as to how to do that: variable playbooks, role inclusion, or inheritance. but 438281 seems like a good start and we can go from there. | 17:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Stop json-encoding the nodepool metadata https://review.openstack.org/297950 | 17:21 |
| jeblair | mordred: and then 410812 is the v3 version. | 17:23 |
| mordred | jeblair: nod | 17:23 |
| Shrews | mordred: hrm, could you rebase 410812 on 443690, lest we conflict | 17:24 |
| Shrews | ? | 17:24 |
| mordred | Shrews: sho nuff! | 17:24 |
| Shrews | gym time. bbl | 17:26 |
| *** yolanda has joined #zuul | 17:39 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 17:44 |
| *** hashar has quit IRC | 17:55 | |
| *** bhavik1 has joined #zuul | 18:00 | |
| *** bhavik1 has quit IRC | 18:01 | |
| *** abregman|mtg is now known as abregman|afk | 18:05 | |
| *** Cibo_ has joined #zuul | 18:19 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'push' and 'tag' github webhook events. https://review.openstack.org/443947 | 19:34 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Remove old/dead classes https://review.openstack.org/443644 | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add leaked instance cleanup https://review.openstack.org/443690 | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Add back statsd reporting https://review.openstack.org/443605 | 20:02 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Stop json-encoding the nodepool metadata https://review.openstack.org/410812 | 20:02 |
| Shrews | jeblair: sorry, the bottom of that stack (443605) had a bug using the wrong variable name | 20:03 |
| Shrews | mordred: rebased yours for you as well | 20:03 |
| *** hashar has joined #zuul | 20:07 | |
| mordred | Shrews: yay thanks! | 20:20 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'push' and 'tag' github webhook events. https://review.openstack.org/443947 | 20:22 |
| mordred | jeblair: I had this idea the other day which is almost certainly terrible, but jlk's patch there makes me think of it again ... | 20:23 |
| mordred | jeblair: which is some sort of sphinx plugin similar to the jjb one that reads info from the various plugins and produces the documentation additions for how to configure/use them | 20:24 |
| jlk | allowing driver doc to live in the driver tree instead of everybody piling on the same docs files? | 20:24 |
| mordred | yah | 20:31 |
| mordred | well - and also, some of the things I'm watching be added in places seem like they could actually be documented by introspecting the source smartly | 20:31 |
| mordred | (I had the thought initially related to the sql reporter plugin) | 20:32 |
| jlk | oh yeah. | 20:32 |
| jlk | although, "I have a problem with docs, so I added sphynx. Now I have two problems with docs." | 20:33 |
| jlk | it's kind of like fighting with HTML WYSIWYG editors, it'l generate something approximately like what you want, but getting it exactly how you want will be a never ending nightmare. | 20:34 |
| Shrews | SpamapS: lulz https://github.com/Netflix/zuul/issues/315 | 20:39 |
| SpamapS | Shrews: hopefully they'll consider just renaming themselves. :) | 20:40 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'pr-comment' github webhook event https://review.openstack.org/443959 | 20:42 |
| mordred | SpamapS: wow. nice | 20:50 |
| Shrews | mordred: question about 410812 | 21:05 |
| Shrews | mordred: before your change, we conditionally added values to the metadata if that data was supplied | 21:06 |
| Shrews | mordred: your change always puts something into metadata | 21:06 |
| Shrews | trying to decide if that could be harmful somewhere... | 21:07 |
| Shrews | i know we don't ever supply node_id now, nor do we look for it. | 21:07 |
| Shrews | not sure about the others | 21:07 |
| Shrews | but it passes tests, so... meh? | 21:09 |
| tflink | pabelanger: would you be interested in feedback on the .zuul.yml format reviews from fedora and redhat automation folk? | 21:09 |
| Shrews | ooh, it doesn't pass the devstack tests. | 21:10 |
| pabelanger | tflink: indeed, I think anybody interested in zuulv3 should be happy to provide feedback | 21:11 |
| tflink | ok, just wanted to make sure before looking more closely and thinking about commenting | 21:12 |
| rbergeron | tflink!! | 21:16 |
| rbergeron | yesplz | 21:17 |
| tflink | rbergeron: I sent the reviews to _ari_ as well. it'd be nice if these groups could be aiming to use a similar, if not identical format | 21:18 |
| rbergeron | spamaps: nice issue in netflix ;) | 21:18 |
| rbergeron | tflink: indeed. or at least, yiu know, discussing and sharing | 21:19 |
| rbergeron | since use cases are nice | 21:19 |
| rbergeron | and now, my plane finally departs | 21:20 |
| tflink | rbergeron: safe travels | 21:22 |
| rbergeron | tflink: thanks! safe... meeeeeetings? ;) | 21:30 |
| tflink | thanks | 21:39 |
| tflink | they seem to be done for the day so I should be safe-ish :) | 21:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Improve job dependencies using graph instead of tree https://review.openstack.org/443973 | 21:51 |
| jeblair | that is going to fail most of the unit tests because of the significant syntax change | 21:52 |
| jeblair | however, it passes the 'test_jobs_launched' test, as well as its own additional tests | 21:52 |
| jeblair | so i think it's in good enough shape to pause for a moment while we check out how it looks and decide if that's the way forward | 21:53 |
| jeblair | mordred, pabelanger, clarkb, SpamapS: ^ | 21:53 |
| *** abregman|afk has quit IRC | 22:08 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add 'pr-comment' github webhook event https://review.openstack.org/443959 | 22:15 |
| jeblair | 22:19 < openstackgerrit> James E. Blair proposed openstack-infra/infra-specs master: Zuul v3: Silly whitespace fixes https://review.openstack.org/443983 | 22:19 |
| jeblair | 22:19 < openstackgerrit> James E. Blair proposed openstack-infra/infra-specs master: Zuul v3: remove references to swift https://review.openstack.org/443984 | 22:19 |
| jeblair | 22:19 < openstackgerrit> James E. Blair proposed openstack-infra/infra-specs master: Zuul v3: update job trees to graphs https://review.openstack.org/443985 | 22:19 |
| jeblair | those may be of interest to the channel | 22:19 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/zuul master: Only depend-on open changes https://review.openstack.org/254957 | 22:34 |
| _ari_ | tflink: +1 | 22:38 |
| *** jamielennox is now known as jamielennox|away | 22:42 | |
| *** jamielennox|away is now known as jamielennox | 22:45 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Configurable SSH access to GitHub https://review.openstack.org/444034 | 22:56 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/zuul master: Only depend-on open changes https://review.openstack.org/254957 | 23:05 |
| SpamapS | jeblair: you broke the tests out into a list and I totally forgot where that was. Can you remind me? Thanks. | 23:22 |
| jeblair | SpamapS: https://etherpad.openstack.org/p/zuulv3skips | 23:25 |
| *** saneax-_-|AFK is now known as saneax | 23:25 | |
| SpamapS | jeblair: ty | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!