| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Pin async-timeout https://review.openstack.org/566488 | 01:13 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/nodepool master: Remove initial stub release note https://review.openstack.org/566409 | 01:46 |
| *** EvilienM is now known as EmilienM | 02:45 | |
| *** threestrands has joined #zuul | 04:04 | |
| *** sshnaidm has joined #zuul | 04:17 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul master: Add tox-py36 jobs https://review.openstack.org/565881 | 05:08 |
| *** sshnaidm has quit IRC | 05:14 | |
| *** pcaruana has joined #zuul | 06:02 | |
| *** jesusaur has quit IRC | 06:05 | |
| *** jesusaur has joined #zuul | 06:06 | |
| *** tflink has quit IRC | 06:16 | |
| *** AJaeger_away has quit IRC | 06:21 | |
| *** AJaeger has joined #zuul | 06:28 | |
| *** tflink has joined #zuul | 06:38 | |
| *** gtema has joined #zuul | 06:43 | |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Add role information to task in zuul_json callback https://review.openstack.org/559998 | 06:43 |
| openstackgerrit | Benedikt Löffler proposed openstack-infra/zuul master: Add role information to task in zuul_json callback https://review.openstack.org/559998 | 06:44 |
| *** AJaeger has quit IRC | 06:45 | |
| *** AJaeger has joined #zuul | 06:52 | |
| *** toabctl has quit IRC | 07:18 | |
| *** toabctl has joined #zuul | 07:22 | |
| *** threestrands has quit IRC | 07:39 | |
| *** jpena|off is now known as jpena | 07:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Remove use of six https://review.openstack.org/566159 | 08:16 |
| *** sshnaidm has joined #zuul | 09:35 | |
| *** sshnaidm is now known as sshnaidm|rover | 09:35 | |
| *** yolanda_ is now known as yolanda | 10:35 | |
| openstackgerrit | Artem Goncharov proposed openstack-infra/nodepool master: fail to build image without known type https://review.openstack.org/566437 | 10:45 |
| *** jpena is now known as jpena|lunch | 11:32 | |
| *** elyezer has joined #zuul | 11:46 | |
| *** jpena|lunch is now known as jpena | 12:28 | |
| *** rlandy has joined #zuul | 12:35 | |
| *** dkranz has quit IRC | 12:53 | |
| *** mmedvede has quit IRC | 12:58 | |
| *** zigo_ has joined #zuul | 13:07 | |
| *** adam_g_ has joined #zuul | 13:09 | |
| *** sdake_ has joined #zuul | 13:09 | |
| *** sdake_ has joined #zuul | 13:09 | |
| *** sdake has quit IRC | 13:11 | |
| *** openstackgerrit has quit IRC | 13:11 | |
| *** rcarrillocruz has quit IRC | 13:11 | |
| *** zigo has quit IRC | 13:11 | |
| *** adam_g has quit IRC | 13:11 | |
| *** adam_g_ is now known as adam_g | 13:11 | |
| mnaser | morning, would it be possible to have some eyes on https://review.openstack.org/#/c/566488 ? | 13:15 |
| *** rcarrillocruz has joined #zuul | 13:18 | |
| *** openstackgerrit has joined #zuul | 13:24 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Add logo to docs https://review.openstack.org/566410 | 13:24 |
| *** gtema has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Switch to stestr https://review.openstack.org/536862 | 13:24 |
| *** mmedvede has joined #zuul | 13:26 | |
| openstackgerrit | Wojciech Urbanski proposed openstack-infra/zuul-jobs master: Add RedHat support for role: configure-mirrors https://review.openstack.org/566580 | 13:29 |
| openstackgerrit | Wojciech Urbanski proposed openstack-infra/zuul-jobs master: Add RedHat support for role: configure-mirrors https://review.openstack.org/566580 | 13:30 |
| *** dkranz has joined #zuul | 13:51 | |
| openstackgerrit | Merged openstack-infra/zuul master: Pin async-timeout https://review.openstack.org/566488 | 14:06 |
| *** jimi|ansible has quit IRC | 14:16 | |
| *** yolanda has quit IRC | 14:38 | |
| *** yolanda has joined #zuul | 14:45 | |
| *** pcaruana has quit IRC | 15:09 | |
| openstackgerrit | Fatih Degirmenci proposed openstack-infra/zuul master: Add additional steps for configuring Zuul services on CentOS 7 https://review.openstack.org/565075 | 15:29 |
| *** jpena is now known as jpena|brb | 15:46 | |
| *** openstackgerrit has quit IRC | 16:19 | |
| *** jpena|brb is now known as jpena | 16:23 | |
| *** openstackgerrit has joined #zuul | 16:38 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Ignore .stestr directory in .gitignore https://review.openstack.org/566676 | 16:38 |
| Shrews | Easy review there ^^ | 16:39 |
| clarkb | Shrews: I just quickly single core approved it since it is mostly trivial and will address local test user ui problems | 16:40 |
| Shrews | clarkb: yeah, danke | 16:40 |
| openstackgerrit | Merged openstack-infra/zuul master: Add logo to docs https://review.openstack.org/566406 | 16:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Only emit parent-change-enqueued if needed https://review.openstack.org/563194 | 17:10 |
| *** jpena is now known as jpena|off | 17:21 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Ignore .stestr directory in .gitignore https://review.openstack.org/566676 | 18:08 |
| *** nguyenhai_ has joined #zuul | 18:09 | |
| *** nguyenhai has quit IRC | 18:12 | |
| corvus | Shrews, clarkb, mordred: i've ciphered on the multiple label question over the weekend, and i believe that we should continue to go down the path of nodes supporting multiple labels. i think that ultimately we will need something like that (our conversation with fdegir showed that while we can achieve quite a lot with the features we have now, we're ultimately going to run into situations that require some | 18:33 |
| corvus | form of flexibility with labels, and probably sooner rather than later). i think clarkb's idea of supporting multiple label requests is good, but i think it's going to be a better ux to be able to specify the single thing you need, that way you don't have to keep updating jobs as labels change. maybe we'll end up adding that eventually, but for now, i think the focus should be on nodes offering multiple | 18:33 |
| corvus | labels. | 18:33 |
| corvus | Shrews, clarkb, mordred: if you agree, i think the next steps are probably to continue to identify what will need to change -- so far we've identified that some of the statsd reporting will have to be dropped or its meaning altered (and presumably the admin listing tables as well), and we need to address what it means for ready nodes. | 18:34 |
| Shrews | corvus: i've also been thinking about this. It's going to change the way we make hostnames for openstack nodes (we use label in the hostname). I'm currently thinking about how this might affect configuration processing | 18:35 |
| clarkb | Shrews: for that particular case probably want to make it more image based rather than label based? | 18:35 |
| clarkb | so use the image name regardless of what the label is for that stuff since it should be unique? | 18:36 |
| corvus | Shrews: yep! there's another one. :) i hope that we (openstack) don't use that for anything important at this point; if we do, we shouldn't. :) | 18:36 |
| Shrews | clarkb: possibly. corvus, maybe we can etherpad a sample nodepool.yaml config file using the openstack driver so we can agree on how this might look and work | 18:37 |
| corvus | Shrews, clarkb, mordred: so it'd be good to continue further down the road and find any more 'gotchas' like that. then we can make a final call as to whether the change is worth it. granted, that probably means mostly implementing the change before we find them all, but, at least so far, i'm inclined to think we'll end up saying 'yes' at the end of the road. | 18:37 |
| corvus | Shrews: excellent idea | 18:37 |
| Shrews | corvus: the only way i'm finding these "gotchas" is just going ahead with implementation. stuck in the config stuff now :/ | 18:38 |
| corvus | Shrews: i've been trying to come up with a concise way of stating what i think should be the same resolution to both the stats and the ready-node problem: "labels should still be treated independently". so if a node has 2 labels, then we'd emit a stat for both labels, and it would contribute toward the ready-node count for both labels. and when it became used, it would decrease the ready-count for both | 18:39 |
| corvus | labels, and probably, i guess, increase the used count for only the label it is satisfying. | 18:39 |
| corvus | Shrews: i'm hoping that sort of thing helps us craft a consistent answer to questions like that | 18:39 |
| Shrews | corvus: we can't just increase the used count on the satisfied label. that would imply the unused label(s) would still show as 'ready' and cause confusion, yeah? | 18:41 |
| clarkb | do we separate track total nodes in a way that allows us to get an aggregate that is accurate? | 18:41 |
| clarkb | might need to add ^ if not | 18:41 |
| Shrews | e.g, we have 1 total node with labels "ubuntu-xenial" and "ubuntu-latest". a job uses ubuntu-xenial. | 18:41 |
| Shrews | the graph would show ubuntu-latest as available and ready | 18:42 |
| Shrews | that seems... weird | 18:42 |
| Shrews | but then, i don't use these stats, so i defer to those that do | 18:42 |
| corvus | Shrews: sorry i wasn't clear, i meant in that case, the graph should then show zero ubuntu-xenial and zero ubuntu-latest ready nodes, and one ubuntu-xenial used node. | 18:42 |
| clarkb | They are mostly used for why are jobs not starting I think to quickly identify why things might be queued | 18:43 |
| Shrews | ah | 18:43 |
| clarkb | which would work in this scenario | 18:43 |
| corvus | clarkb: i think a consequence of this is that you will not be able to track total "real" nodes by label, but you can still do so by provider. | 18:43 |
| corvus | clarkb: and exactly -- the label stats then show you why no jobs are starting | 18:43 |
| corvus | (but they don't show you that you're at quota) | 18:43 |
| corvus | for example, looking at the way we (openstack) tracks usage, that wouldn't be a significant change for us -- we generally track capacity by provider, and our per-label graphs are constructed in a way that doesn't lend itself to summing different node types, or comparing them to capacity. | 18:47 |
| Shrews | corvus: clarkb: maybe we can start with this yaml from one of the nodepool tests: https://etherpad.openstack.org/p/multi-label-nodepool-config | 18:47 |
| Shrews | i'm assuming the multiple labels would be in the "pool" section? | 18:48 |
| Shrews | outer label would still need to be singlely named? | 18:48 |
| corvus | Shrews: i believe so | 18:49 |
| corvus | Shrews: i mocked up the fake-label4 option with multiple labels based on the closest pattern i could think of to the existing support in the static driver | 18:50 |
| Shrews | and what if one of the referenced labels does not exist? | 18:50 |
| corvus | Shrews: i think we still need them all to exist (that's true for static too, right?) | 18:51 |
| Shrews | corvus: i think the static driver just matches the requested label to any of the supported labels. i don't *think* they all need to exist, but i haven't considered this until just now | 18:52 |
| clarkb | instead have having name be a list why not use the existing provider.labels list and allow duplicates with different images/ram/flavor/etc | 18:52 |
| clarkb | I guess it wouldn't be a duplicate we'd just allow multiple labels in that list to have the same image | 18:53 |
| corvus | Shrews: hrm, that may be an oversight in the static driver -- if they're required for the dynamic ones, i don't see why they shouldn't be for static... | 18:53 |
| corvus | clarkb: yeah, that's already the case. you could argue that means we don't *really* need to do anything here. | 18:53 |
| Shrews | corvus: i'm on board with that. just want to establish what we should do | 18:54 |
| corvus | clarkb: since, in the context of a dynamic provider, a label is basically not much more than an image+ram combination | 18:55 |
| corvus | er, image+flavor | 18:55 |
| corvus | clarkb: i guess the thing there is that the *nodes* need to support multiple labels | 18:56 |
| corvus | clarkb: so if we spin up an ubuntu-xenial node to satisfy min-ready, it ought to have both an ubuntu-xenial label as well as ubuntu-latest label, in order for the most sensible thing to happen when a request comes in. | 18:57 |
| Shrews | wait | 18:58 |
| corvus | clarkb: so the question becomes, do we make that explicit like in the configuration in the etherpad, or implicit by having nodepool match up duplicate label entries. | 18:58 |
| corvus | Shrews: waiting :) | 18:58 |
| Shrews | yeah, not sure how we'd apply both labels to a min-ready node when that basically comes from the outer labels | 18:59 |
| Shrews | there's no mapping there | 18:59 |
| corvus | Shrews: well, when the node is created, it's done from a particular pool, so (if we go with the example in the etherpad) we know all the labels it will end up with. | 18:59 |
| clarkb | and it satisfies all of them at the same time | 19:00 |
| clarkb | but once used it no longer satisfies any of them | 19:00 |
| Shrews | corvus: unless multiple matches are found in the pool | 19:00 |
| corvus | Shrews: that's a good point | 19:00 |
| Shrews | eg., "ubuntu-xenial/networking" and a "ubuntu-xenial/ubuntu-latest" | 19:00 |
| Shrews | so a) do we allow that? | 19:01 |
| Shrews | b) if we do, uh... | 19:01 |
| corvus | (a) it seems like it should be allowed (if we do this); (b) uh... indeed... i guess we would need to pick one at random or pick the first? | 19:02 |
| Shrews | "random" could leave the request unsatisfied until it gets the right thing | 19:03 |
| Shrews | maybe? this is hurting my brain | 19:03 |
| Shrews | i guess not | 19:04 |
| corvus | Shrews: well, any thing which matches should satisfy it. | 19:04 |
| Shrews | but i can now see someone wanting to do min-ready by label combinations | 19:04 |
| clarkb | the node requests are always for a specific lable right? | 19:04 |
| corvus | there's also a pretty good chance that if you put "ubuntu-xenial: min-ready: 1" and "ubuntu-latest: min-ready: 1", you'll end up with 2 ready nodes, because while the algorithm wouldn't add any new ones if one already existed, it wouldn't know that it could satisfy both with only one node until it was actually created. so that would sometimes happen, especially on startup or when things were busy. | 19:04 |
| clarkb | so you'd assign a node matching that lable to the request, remove it from valid list of any other labels it satisfies then return it to zuul | 19:05 |
| clarkb | oh you mean on the front end of booting the min ready nodes | 19:05 |
| corvus | Shrews: well, the goal here is not to support label combinations either in min-ready or in zuul. you still only request one specific thing. | 19:05 |
| clarkb | I thought it was on the request side pulling them out | 19:05 |
| corvus | clarkb: both really -- min-ready is still handled by requests; it's just an automatic request. | 19:06 |
| Shrews | yeah, min-ready will get even _less_ predictable, but i don't see a resolution to that | 19:07 |
| Shrews | (folks already complain about it) | 19:07 |
| corvus | Shrews: what's the complaint? | 19:07 |
| Shrews | just that you can get more than min-ready count | 19:08 |
| Shrews | it's a timing issue | 19:08 |
| corvus | ah. well, that's right there in the name. :) | 19:08 |
| Shrews | exactly | 19:08 |
| Shrews | we can just move even farther past "min" now :) | 19:09 |
| corvus | yep | 19:09 |
| Shrews | we have sooo much code that indexes dicts by label names. this will be a "fun" change | 19:09 |
| clarkb | we could "merge" the labels and only do min ready requests for the aggregate sets | 19:09 |
| clarkb | so ubuntu-xenial + special-label2 would result in a single min ready request for one of the two if min-ready is set to 1 for both labels | 19:10 |
| corvus | anyway, i think the etherpad example is workable, with all the caveats that we've discussed. but also, it's probably worth considering whether this feels too weird for the dynamic drivers, and instead we should just focus on fundamental support for multiple labels, but still only have it implemented for the static driver. we could even do that, and still leave implementation for dynamic drivers as a future | 19:10 |
| corvus | change, if we feel it's useful. | 19:10 |
| corvus | clarkb: that kind of merging might produce different results in different provider-pools, so you'd end up limiting the places you get your min-ready nodes from | 19:11 |
| Shrews | clarkb: that won't always work b/c the new node could grab a label that isn't the correct combination | 19:11 |
| clarkb | corvus: thats a good point, not sure I understand Shrews' concern | 19:12 |
| clarkb | Shrews: if you made a new node for ubuntu-xenial but it also satisfied ubuntu then a request for ubuntu-xenial or ubuntu could be satisfied by that node | 19:12 |
| corvus | i think Shrews concern is that if min-ready requests 'ubuntu-xenial' because it thinks it's going to also get 'spceial', it might end up with the one place in the system where it gets 'ubuntu-xenial+notspecial' | 19:13 |
| Shrews | clarkb: since we'd allow labels to be duplicated, the new node might not be labeled with "ubuntu/ubuntu-xenial", but instead "ubuntu/networking", leaving ubuntu-xenial unsatisfied | 19:13 |
| Shrews | what corvus said | 19:13 |
| clarkb | because this is per provider, got it | 19:14 |
| Shrews | corvus: clarkb: I'll try to summarize this in a response to my original email (mostly so I don't forget it all) | 19:17 |
| corvus | Shrews: cool, thanks! | 19:17 |
| corvus | i will get lunch now | 19:18 |
| clarkb | could (should) that be global config and report an error if a new launcher starts with a config that doesn't line up? | 19:18 |
| Shrews | clarkb: not following | 19:20 |
| clarkb | Shrews: if a provider label matches one label in the set it implicitly must match all of them | 19:21 |
| clarkb | or explicitly has to and if not is an error | 19:21 |
| clarkb | (we'd probably have to put more data in zk and query for it | 19:21 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Split logging of inventory to its own role https://review.openstack.org/563787 | 19:37 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Run authorized_keys as root https://review.openstack.org/563889 | 20:05 |
| *** acozine1 has joined #zuul | 20:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-base-jobs master: Add logo to docs https://review.openstack.org/566412 | 20:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-base-jobs master: Add libre2 to bindep https://review.openstack.org/566731 | 20:39 |
| *** zhuli_ has joined #zuul | 20:45 | |
| *** acozine2 has joined #zuul | 20:47 | |
| *** acozine1 has quit IRC | 20:53 | |
| *** zhuli has quit IRC | 20:53 | |
| *** zhuli_ is now known as zhuli | 20:53 | |
| *** dkranz has quit IRC | 21:00 | |
| *** acozine2 has quit IRC | 21:29 | |
| -openstackstatus- NOTICE: Any devstack job failure due to rsync errors related to tripleo-incubator can safely be rechecked now | 22:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Remove use of six https://review.openstack.org/566159 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!