| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 00:07 |
|---|---|---|
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 00:08 |
| *** rlandy is now known as rlandy|afk | 00:13 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add initial GraphQL controller https://review.openstack.org/574625 | 00:57 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: use a declarative base model https://review.openstack.org/575275 | 00:57 |
| *** rlandy|afk is now known as rlandy | 01:10 | |
| *** rlandy has quit IRC | 01:35 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: sql: use a declarative base model https://review.openstack.org/575275 | 03:26 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add initial GraphQL controller https://review.openstack.org/574625 | 03:26 |
| tobiash | corvus, mordred: I just found a simpler way solving this issue without forking the linear strategy | 04:15 |
| tobiash | corvus, mordred: using the task uuid itself works without forking the linear plugin (just tested that locally) | 04:18 |
| tobiash | corvus, mordred: do we really need that host part? Right now the callback is called exactly once for a multi-host task | 04:20 |
| tobiash | corvus, mordred: so that's already now a problem if we use delegate | 04:21 |
| tobiash | corvus, mordred: so thinking about this this was always wrong but I think that should be fixed | 04:22 |
| *** threestrands has quit IRC | 04:47 | |
| *** pcaruana has quit IRC | 05:09 | |
| *** leifmadsen has quit IRC | 05:27 | |
| *** leifmadsen has joined #zuul | 05:28 | |
| *** pcaruana has joined #zuul | 06:17 | |
| *** AJaeger has quit IRC | 06:24 | |
| *** threestrands has joined #zuul | 06:27 | |
| *** hashar has joined #zuul | 06:33 | |
| *** AJaeger has joined #zuul | 06:57 | |
| *** Rohaan has joined #zuul | 06:57 | |
| *** gtema has joined #zuul | 07:01 | |
| *** jpena|off is now known as jpena | 07:30 | |
| *** nguyenhai has quit IRC | 07:37 | |
| *** nguyenhai has joined #zuul | 07:37 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move zuul_log_id injection to command action plugin https://review.openstack.org/575351 | 07:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix log streaming for delegated hosts https://review.openstack.org/575352 | 07:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Revert "Temporarily override Ansible linear strategy" https://review.openstack.org/575353 | 07:38 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove extra argument when logging logger timeout https://review.openstack.org/575354 | 07:38 |
| *** electrofelix has joined #zuul | 08:33 | |
| *** ianychoi has quit IRC | 08:39 | |
| *** flawi has joined #zuul | 09:35 | |
| flawi | hola, is anyone aware of any effort being put into supporting bitbucket hosted repositories in zuul? just wondering if there's something we could help with, or if we should just assume that we'll be writing that integration from scratch if we plan on using zuul | 09:42 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move zuul_log_id injection to command action plugin https://review.openstack.org/575351 | 09:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix log streaming for delegated hosts https://review.openstack.org/575352 | 09:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Revert "Temporarily override Ansible linear strategy" https://review.openstack.org/575353 | 09:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove extra argument when logging logger timeout https://review.openstack.org/575354 | 09:44 |
| *** threestrands has quit IRC | 09:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move zuul_log_id injection to command action plugin https://review.openstack.org/575351 | 10:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix log streaming for delegated hosts https://review.openstack.org/575352 | 10:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Revert "Temporarily override Ansible linear strategy" https://review.openstack.org/575353 | 10:28 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove extra argument when logging logger timeout https://review.openstack.org/575354 | 10:28 |
| tobiash | flawi: I don't know of any but if you're interested you could start with the github driver as template as I think github works at least a bit similar like bitbucket | 10:30 |
| flawi | yeah, should be entirely doable, just making sure we don't duplicate efforts there if someone is already working on it | 10:31 |
| tobiash | flawi: I think I heard the same question with gitlab but at least I didn't notice any serious effort right now | 10:31 |
| flawi | all right, thanks. I'll just mark it down as something we'll need to do using the GitHub event handling as a starting point, the workflows should be similar | 10:35 |
| tobiash | corvus, mordred: that stack above is green now ^ | 10:50 |
| tobiash | corvus, mordred: that implements the task uuid idea and doesn't need to fork the linear plugin | 10:51 |
| tobiash | corvus, mordred: further I have the strong impression that delegated logging of multiple hosts was broken since the beginning whithout us noticing it so that also includes a fix for that | 10:52 |
| *** jpena is now known as jpena|lunch | 10:57 | |
| *** elyezer has joined #zuul | 11:25 | |
| *** rlandy has joined #zuul | 12:01 | |
| *** jpena|lunch is now known as jpena | 12:03 | |
| mordred | tobiash: oh wow - looking forward to reading! | 12:15 |
| pabelanger | are test_job_aborted and test_abandoned_not_timer know flapping tests? http://logs.openstack.org/73/575173/6/check/tox-py36/d0df7d8/testr_results.html.gz | 12:17 |
| mordred | tobiash: it seems like you could remove the command.py action plugin in the other action dir now | 12:26 |
| tobiash | mordred: Do I have both now? | 12:27 |
| * tobiash is afk atm | 12:28 | |
| mordred | tobiash: yah. but no big deal - we can take care of it in a followup | 12:29 |
| mordred | tobiash: nevermind. I had a stale copy of the tree locally | 12:30 |
| tobiash | :) | 12:32 |
| *** myoung|off is now known as myoung | 12:36 | |
| *** Rohaan has quit IRC | 12:39 | |
| pabelanger | tristanC: replied to ML thread | 12:41 |
| *** ianychoi has joined #zuul | 12:46 | |
| tristanC | pabelanger: awesome, thanks! | 12:48 |
| openstackgerrit | Merged openstack-infra/zuul master: Allow zuul_return in untrusted jobs https://review.openstack.org/575173 | 12:51 |
| pabelanger | mordred: tobiash: clarkb: corvus: It seems tristanC has already created an builtset artifact role in SF.io, but believe would be great to land in zuul-jobs, could trouble you for reviews: https://review.openstack.org/530679/ | 12:52 |
| mordred | pabelanger: lgtm | 12:53 |
| *** elyezer has quit IRC | 12:56 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove extra argument when logging logger timeout https://review.openstack.org/575354 | 13:07 |
| tobiash | mordred: fixed commit message ^ | 13:08 |
| *** elyezer has joined #zuul | 14:15 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Collect the coverage report for npm test jobs https://review.openstack.org/570260 | 14:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location https://review.openstack.org/530679 | 15:14 |
| pabelanger | mordred: tristanC: corvus: ^fixes ansible-lint failure gate pipeline found | 15:15 |
| corvus | mordred, clarkb: did we want to have that call about logging? | 15:20 |
| mordred | corvus: I'm not sure we need to anymore with the new stack from tobiash ? | 15:21 |
| corvus | mordred: oh, do you want to shelve the unix socket idea? | 15:21 |
| mordred | corvus: I don't want to shelve it completely - but I believe the new patches mean we don't necessarily need to jump straight on it (although we certainly can if we want to) | 15:22 |
| *** pcaruana has quit IRC | 15:24 | |
| corvus | mordred: okay, i guess we can leave it for when things settle down and you (or someone else) want(s) to pick it up again? i'll go review tobiash's patches now | 15:26 |
| corvus | mordred: your comment on 575351 -- is that the one you decided was erroneous in irc? | 15:27 |
| mordred | corvus: yes | 15:28 |
| tobiash | there was no command.py action override before that change | 15:28 |
| corvus | clarkb: 575351 and children have +2s from mordred and I if you want to review / +W them | 15:34 |
| clarkb | ok I'm trying ti get my windoe manager back then will look | 15:35 |
| mordred | clarkb: tl;dr - I was wrong, you were right | 15:36 |
| clarkb | ok I switched back to desktop and will just not update for a bit here | 15:39 |
| *** hashar is now known as hasharAway | 15:40 | |
| *** sshnaidm has quit IRC | 15:55 | |
| *** myoung is now known as myoung|lunch | 16:04 | |
| clarkb | mordred: corvus tobiash in https://review.openstack.org/#/c/575352/3/zuul/ansible/callback/zuul_stream.py we set host to the delegate to value when delegating but then use the second half of the new tuple return (inventory_hostname) to find the log_id. Does that all work out still with delegated tasks? | 16:07 |
| clarkb | on the action module command side it is still going to use the inventory hostname of the host in the task not the one it was delegated to? | 16:07 |
| corvus | clarkb: can you give a line number for "set host to the delegate to value" ? | 16:10 |
| clarkb | 572 | 16:10 |
| tobiash | clarkb: host is where we need to connect to for log streaming | 16:11 |
| tobiash | clarkb: there is also a test case which is exercising this | 16:12 |
| *** elyezer has quit IRC | 16:12 | |
| clarkb | tobiash: ya, I think my question boils down to is the inventory hostname the delegated to value in the command module side | 16:12 |
| clarkb | since we need the two to match up. Sounds like it isn't if you delegate to the inventory hostname is still whatever node that is "running" on | 16:13 |
| tobiash | clarkb: in the name of the logfile we actually want the inventory hostname | 16:13 |
| tobiash | using the delegated hostname for the logfile would be useless as it's not unique | 16:13 |
| tobiash | clarkb: so we connect to the delegate but request the logfile with the inventory hostname | 16:14 |
| corvus | clarkb: in 575351 we have host = paths._sanitize_filename(task_vars.get('inventory_hostname')) | 16:14 |
| corvus | so yes, the remote side is always going to have "inventory_hostname" in its log path | 16:14 |
| clarkb | oh right, thanks | 16:15 |
| clarkb | ok I've +2'd things but not approved. My allergies are bad enough that I think I may try to take it easy to day | 16:15 |
| clarkb | (so likely won't be around for executor restarts) | 16:15 |
| corvus | i'll approve and do a full restart today so that hopefully we can release today or tomorrow | 16:16 |
| ssbarnea | i receive lots of emails from storyboard about gerrit comments and I am unable to find why I get those as I am not CC neither on the CR nor on the story. | 16:16 |
| ssbarnea | example: https://storyboard.openstack.org/#!/story/2002528 | 16:16 |
| corvus | ssbarnea: did you subscribe to the project? | 16:17 |
| ssbarnea | corvus: yep, this was my "mistake" to press the star on zuul. | 16:17 |
| ssbarnea | thanks, resolved it. too much noise | 16:18 |
| corvus | ssbarnea: https://storyboard.openstack.org/#!/profile/preferences and https://storyboard.openstack.org/#!/project/679 | 16:18 |
| *** sshnaidm has joined #zuul | 16:40 | |
| pabelanger | corvus: do you mind refreshing your +3 on https://review.openstack.org/530679/ new patch addressed ansible-lint failure | 17:09 |
| *** myoung|lunch is now known as myoung | 17:12 | |
| *** jpena is now known as jpena|off | 17:14 | |
| corvus | pabelanger: where's the failure? | 17:17 |
| pabelanger | corvus: it never got reported back to gerrit as I pushed up new patch, but http://logs.openstack.org/79/530679/8/gate/openstack-zuul-jobs-linters/7444209/job-output.txt.gz is the log | 17:19 |
| corvus | pabelanger: gotcha, i +3d | 17:19 |
| pabelanger | thanks | 17:19 |
| Shrews | pabelanger: corvus: i believe i found a nodepool bug that may cause the node leak. working up a test case | 17:24 |
| pabelanger | Shrews: yay | 17:26 |
| pabelanger | speaking of nodepool, should region-name be fetched from clouds.yaml over nodepool.yaml? Have a case in rdo nodepool, we don't set region-name, but nodepool.region in inventory ends up null: https://logs.rdoproject.org/01/14201/2/check/legacy-DLRN-rpmbuild/8435496/zuul-info/inventory.yaml | 17:30 |
| pabelanger | actually, I think it does work like that. seems we don't have it set in our clouds.yaml right now | 17:32 |
| pabelanger | testing | 17:32 |
| corvus | looks like 575351 hit a timeout on test_plugins. i'll look at that locally but re-enqueue the stack (which otherwise passed all tests) | 17:38 |
| corvus | wow, test_plugins takes 31 seconds on my machine. we might want to bump that timeout to 120 | 17:43 |
| *** gtema has quit IRC | 17:43 | |
| *** pcaruana has joined #zuul | 17:43 | |
| corvus | oh, it already is... :/ | 17:44 |
| corvus | let's just keep an eye out on it | 17:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix paused handler exception handling https://review.openstack.org/575515 | 17:57 |
| Shrews | I actually have a fix for ^^ but waiting to show that failure before I add it in PS2 | 18:00 |
| *** elyezer has joined #zuul | 18:00 | |
| *** pcaruana has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add buildset-artifacts-location https://review.openstack.org/530679 | 18:06 |
| *** pcaruana has joined #zuul | 18:06 | |
| Shrews | pabelanger: corvus: looks to me like zuul has most of those locks, so i suspect the executor restarts | 18:06 |
| Shrews | 6 of them are part of 2 different requests (3 nodes each request) | 18:08 |
| Shrews | at least 2 of the other 4 are likely due to the bug i found. going to assume the other 2 fall into one of those 2 categories | 18:13 |
| pabelanger | okay, region_name doesn't seem to get pulled from clouds.yaml, if missing from nodepool.yaml | 18:15 |
| *** electrofelix has quit IRC | 18:19 | |
| clarkb | pabelanger: for the single region case? | 18:23 |
| pabelanger | clarkb: yah, in this case we only have regionOne for the cloud, and configs opted not to set the region-name value, but nodepool.region in ansible inventory is null | 18:24 |
| pabelanger | so, proposing patches to set it in nodepool.yaml now, but feel we could also load it from clouds.yaml if set | 18:25 |
| pabelanger | or just remove nodepool.yaml region-name all together and fall back to clouds.yaml? | 18:25 |
| *** sshnaidm is now known as sshnaidm|off | 18:26 | |
| clarkb | you can't remove it from nodepool.yaml because clouds can have more than one region | 18:29 |
| clarkb | the setting there is to select the region from the valid list of regions in clouds.yaml. osc does the same thing | 18:30 |
| clarkb | we could do like osc and error if more than one region is in the clouds.yaml | 18:30 |
| clarkb | not sure if that is more or less friendly than erroring if you don't set it | 18:30 |
| pabelanger | yah, that is true | 18:32 |
| pabelanger | okay, we've proposed a patch to just set region-name in nodepool.yaml | 18:33 |
| clarkb | er error if more than one region is in clouds.yaml and you didn't set on in nodepool config | 18:33 |
| clarkb | its the tradeoff between telling everyone to just set it and that being simple beacuse its one rule vs being smart and having conditionals in your rules but being easier for the simple case | 18:34 |
| openstackgerrit | Merged openstack-infra/zuul master: Move zuul_log_id injection to command action plugin https://review.openstack.org/575351 | 18:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix log streaming for delegated hosts https://review.openstack.org/575352 | 18:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Revert "Temporarily override Ansible linear strategy" https://review.openstack.org/575353 | 18:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove extra argument when logging logger timeout https://review.openstack.org/575354 | 18:36 |
| *** GonZo2000_ has quit IRC | 18:52 | |
| *** dkranz has quit IRC | 19:20 | |
| *** dkranz has joined #zuul | 19:21 | |
| pabelanger | question, could min-ready in nodepool be scoped to provider too vs just label? I can see a case where a nodepool provider might not actually want to idle ready nodes before a job actually needs it, due to cost. However another provider it might be okay | 20:55 |
| pabelanger | I guess that also gets into the idea of provider precedences | 20:58 |
| *** hasharAway has quit IRC | 21:37 | |
| *** myoung is now known as myoung|off | 21:45 | |
| corvus | pabelanger: my guess is it'd be best to try to solve that with a general preference mechanism (because the same reason you wouldn't want one to idle is probably the same reason you wouldn't want to use it in general) | 22:22 |
| fungi | ahh, like having some specified "burst providers" who only provide nodes as a last resort when you're otherwise out of capacity | 22:38 |
| corvus | yep | 22:39 |
| fungi | i could see that if, say, i had my own private cloud but also an account with a public provider. i'd want to utilize my sunk cost resources first and only pay to burst into the public cloud when i'm out of local capacity | 22:39 |
| corvus | enough people have described that use case that i think if we were to focus only on the min-ready nodes that we still wouldn't have solved the actual problem. min-ready nodes are almost exactly the same as any other nodes in that respect (other than they might be expected to run a bit longer) | 22:40 |
| corvus | but if we solve it generally, we've solved it for min-ready too. | 22:40 |
| fungi | completely agree | 22:41 |
| clarkb | one thing that complicates it is distributed launchers like the ones we have | 22:42 |
| *** elyezer has quit IRC | 22:42 | |
| clarkb | (so may need provider state in zk) | 22:42 |
| corvus | yes, it may require more communication about their state. we can totally add that, i just avoided specifying it in the initial algorithm so we could have something simple and (hopefully) robust to start with. | 22:43 |
| corvus | we knew almost nothing about zk at the time :) | 22:43 |
| clarkb | ya I think it was the right decision :) | 22:43 |
| fungi | prudent | 22:43 |
| clarkb | just point it out for anyone looking to implement it | 22:43 |
| pabelanger | corvus: well, we do want to use the resources, just maybe 12hrs of the day. The rest of the time, we'd might want the provider to be drop back to 0 to avoid billing. But, another cloud maybe donated in this context, which we are happy to keep min-ready nodes | 22:53 |
| pabelanger | but yes, would be happy to discuss a use case more, maybe ptg | 22:54 |
| *** threestrands has joined #zuul | 22:56 | |
| *** threestrands has quit IRC | 22:56 | |
| *** threestrands has joined #zuul | 22:56 | |
| *** threestrands has quit IRC | 22:57 | |
| *** threestrands has joined #zuul | 22:57 | |
| *** threestrands has quit IRC | 22:57 | |
| *** threestrands has joined #zuul | 22:57 | |
| *** GonZo2000_ has joined #zuul | 23:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add supercedent pipeline manager https://review.openstack.org/571932 | 23:32 |
| corvus | clarkb, fungi, tobiash: ^ comments (mostly) addressed | 23:32 |
| *** GonZo2000_ has quit IRC | 23:47 | |
| *** GonZo2000_ has joined #zuul | 23:55 | |
| *** GonZo2000_ has quit IRC | 23:56 | |
| *** GonZo2000_ has joined #zuul | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!