| *** jamesmcarthur has quit IRC | 00:16 | |
| *** jamesmcarthur has joined #zuul | 00:21 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** jamesmcarthur has joined #zuul | 01:03 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #zuul | 01:41 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** jamesmcarthur has joined #zuul | 02:05 | |
| *** ianychoi has quit IRC | 02:19 | |
| *** ianychoi has joined #zuul | 02:21 | |
| *** jamesmcarthur has quit IRC | 03:26 | |
| *** bhavikdbavishi has joined #zuul | 03:52 | |
| *** bhavikdbavishi1 has joined #zuul | 03:54 | |
| *** bhavikdbavishi has quit IRC | 03:56 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:56 | |
| *** bolg has joined #zuul | 04:06 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 04:32 |
|---|---|---|
| *** bhavikdbavishi has quit IRC | 04:41 | |
| *** raukadah is now known as chandankumar | 04:47 | |
| *** wznoinsk has joined #zuul | 04:51 | |
| *** bhavikdbavishi has joined #zuul | 04:55 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:06 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:06 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:22 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:22 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:31 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:31 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:38 |
| *** sshnaidm_ has joined #zuul | 05:59 | |
| *** sshnaidm|pto has quit IRC | 06:00 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:02 |
| *** sshnaidm has joined #zuul | 06:03 | |
| *** sshnaidm_ has quit IRC | 06:04 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 06:05 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:05 |
| *** sshnaidm_ has joined #zuul | 06:07 | |
| *** sshnaidm has quit IRC | 06:10 | |
| *** igordc has quit IRC | 06:10 | |
| *** yolanda__ has quit IRC | 06:20 | |
| *** yolanda has joined #zuul | 06:20 | |
| *** pcaruana has joined #zuul | 06:30 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:34 |
| *** saneax has joined #zuul | 06:42 | |
| *** jpena|off is now known as jpena | 07:00 | |
| *** avass has joined #zuul | 07:04 | |
| *** jangutter_ has joined #zuul | 07:05 | |
| *** jangutter has quit IRC | 07:08 | |
| avass | tobiash: I saw your comment on my change about doing case insensitive matching for labels. I'm guessing there's no case sensitive check when setting labels becuase it just tries to set it and checks the response from gerrit? | 07:13 |
| *** tosky has joined #zuul | 07:19 | |
| tobiash | avass: correct | 07:19 |
| *** jangutter has joined #zuul | 07:22 | |
| avass | tobiash: this part also looked strange to me 'change.missing_labels < set(allow_needs)', is there a reason why it's a prober subset? | 07:23 |
| avass | https://review.opendev.org/gitweb?p=zuul/zuul.git;a=blob;f=zuul/driver/gerrit/gerritconnection.py;h=03e948a61147aa559f524cd4523f51984802c5be;hb=b768ece2c0ecd235c418fe910b84ff88f69860d6#l885 | 07:23 |
| tobiash | Allow_needs is thenset of labels the gate would set itself | 07:25 |
| *** jangutter_ has quit IRC | 07:25 | |
| tobiash | If the set of missing labels (required to merge) is a subset it will be possible to merge after the gate | 07:26 |
| avass | tobiash: what's the content of allow_needs then? the labels we will set in the pipeline? | 07:27 |
| tobiash | Yes | 07:28 |
| avass | tobiash: maybe I'm misunderstanding something then because that looks to me like if missing_labels == allow_needs then it should be able to merge | 07:30 |
| *** hashar has joined #zuul | 07:31 | |
| tobiash | yes, but missing labels on the change can also be less which is ok | 07:31 |
| tobiash | Hence the set operation | 07:32 |
| avass | tobiash: yeah that's what I mean, shouldn't it be a '<=' operator and not a '<'? | 07:32 |
| tobiash | ah now I get what you mean | 07:36 |
| *** gtema has joined #zuul | 07:37 | |
| tobiash | I guess you're right | 07:37 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: config-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:38 |
| tobiash | But I'm wondering how on earth that worked for years as missing_labels == allow_needs sound to me like the most common case | 07:39 |
| avass | tobiash: yep, that's what confuses me | 07:42 |
| avass | tobiash: I'll push a change to gerrit for it | 07:42 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: fix mirror_fqdn variable https://review.opendev.org/688468 | 07:49 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:50 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:51 |
| AJaeger | zbr: did you see ianw's changes https://review.opendev.org/#/c/688616/ ? | 07:53 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 07:53 |
| zbr | AJaeger: nope. | 07:54 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch https://review.opendev.org/644927 | 07:58 |
| zbr | AJaeger: but I am ok with it, I can easily redo my other changes to fit that one. lets merge it. | 07:59 |
| AJaeger | zbr: sure, will review later if nobody beats me to it ;) | 08:04 |
| zbr | sure, sadly i can only do a +1 and ping others to look at it. | 08:10 |
| *** themroc has joined #zuul | 08:20 | |
| *** brendangalloway has joined #zuul | 08:22 | |
| zbr | clarkb: corvus : please r/w ian's two changes for configure-mirrors: https://review.opendev.org/#/c/688616/ (centos8) thanks. | 08:25 |
| *** jamesmcarthur has joined #zuul | 08:30 | |
| *** jamesmcarthur has quit IRC | 08:34 | |
| *** themr0c has joined #zuul | 08:48 | |
| *** themroc has quit IRC | 08:50 | |
| *** bolg has quit IRC | 09:11 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add nodesets API route https://review.opendev.org/605877 | 09:11 |
| *** brendangalloway has quit IRC | 09:15 | |
| *** brendangalloway has joined #zuul | 09:17 | |
| *** sshnaidm_ is now known as sshnaidm|pto | 09:27 | |
| *** themr0c has quit IRC | 09:31 | |
| *** themroc has joined #zuul | 09:32 | |
| avass | tobiash: For consistency with case sensitivity, would it make sense to check labels before setting them? | 10:30 |
| *** bhavikdbavishi has quit IRC | 10:31 | |
| *** gtema has quit IRC | 10:32 | |
| *** themroc has quit IRC | 10:33 | |
| *** themroc has joined #zuul | 10:34 | |
| tobiash | avass: that would be a runtime overhead | 10:36 |
| avass | yeah that's what my thought was. | 10:38 |
| *** recheck has quit IRC | 11:00 | |
| *** recheck has joined #zuul | 11:00 | |
| *** recheck has quit IRC | 11:03 | |
| *** recheck has joined #zuul | 11:04 | |
| *** recheck has quit IRC | 11:05 | |
| *** recheck has joined #zuul | 11:05 | |
| *** recheck has quit IRC | 11:06 | |
| *** recheck has joined #zuul | 11:06 | |
| *** recheck has quit IRC | 11:06 | |
| *** recheck has joined #zuul | 11:07 | |
| *** recheck has quit IRC | 11:08 | |
| *** recheck has joined #zuul | 11:08 | |
| *** recheck has quit IRC | 11:11 | |
| *** recheck has joined #zuul | 11:11 | |
| *** saneax has quit IRC | 11:24 | |
| *** bolg has joined #zuul | 11:38 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| *** avass has quit IRC | 12:00 | |
| *** avass has joined #zuul | 12:03 | |
| *** saneax has joined #zuul | 12:09 | |
| *** brendangalloway has quit IRC | 12:17 | |
| *** paladox has quit IRC | 12:23 | |
| *** paladox has joined #zuul | 12:24 | |
| *** themr0c has joined #zuul | 12:34 | |
| *** themroc has quit IRC | 12:34 | |
| *** saneax has quit IRC | 12:38 | |
| *** jpena|lunch is now known as jpena | 12:41 | |
| recheck | [so-zuul] I want to set spring.profile.active before the zuulServlet is registered → https://stackoverflow.com/questions/58394973/i-want-to-set-spring-profile-active-before-the-zuulservlet-is-registered | 12:54 |
| Shrews | recheck: this is the channel for Zuul the CI/CD project, not the Netflix zuul | 12:55 |
| Shrews | why that question is labeled with openstack-zuul is beyond me | 12:56 |
| *** panda|off is now known as panda | 13:01 | |
| *** brendangalloway has joined #zuul | 13:05 | |
| openstackgerrit | Merged zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 13:16 |
| Shrews | Can someone review/approve https://review.opendev.org/687965 please? | 13:23 |
| *** jangutter_ has joined #zuul | 13:24 | |
| *** jangutter_ has quit IRC | 13:24 | |
| *** jangutter has quit IRC | 13:27 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add the process environment to zuul.conf parser https://review.opendev.org/612824 | 13:28 |
| *** pcaruana has quit IRC | 13:32 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add the process environment to zuul.conf parser https://review.opendev.org/612824 | 13:35 |
| *** spsurya has joined #zuul | 13:42 | |
| tobiash | zuul-maint: interesting, I got three merge failed messages for one recheck of https://review.opendev.org/659991 which is in merge conflict | 13:43 |
| tristanC | could it be because zuul/zuul is registered in both openstack and zuul tenant? | 13:45 |
| fungi | zbr: is "recheck" an irc bot you're running? if so, might be nice to make that clear before folks start getting into conversations with it. (also adding a vocal bot to a channel without agreement of the other channel regulars could be considered rude) | 13:48 |
| zbr | fungi: yeah, is a bot i am running. do you want me to disable it? | 13:52 |
| zbr | i guess the stackoverflow feed is too prone to low quality posts | 13:52 |
| fungi | zbr: i don't feel strongly about it, but will let others weigh in on how useful they do/don't find it | 13:52 |
| fungi | tristanC: that's an interesting theory, but then why three merge failure messages and not two | 13:53 |
| zbr | fungi: is based on rss-feeds, see config at https://github.com/ssbarnea/harem/blob/master/apps/rss-bot/config.yaml#L74-L78 | 13:54 |
| tristanC | fungi: i missed the opendev tenant that also include zuul/zuul | 13:54 |
| *** jamesmcarthur has joined #zuul | 13:55 | |
| zbr | fungi: can you help with https://review.opendev.org/#/c/688616/ ? (and its child) | 13:56 |
| fungi | tristanC: oh, yep, zuul/zuul(:|$) finds three entries in main.yaml | 13:56 |
| fungi | i was initially just searching for zuul/zuul$ | 13:57 |
| fungi | so that seems increasingly likely | 13:57 |
| fungi | zbr: looking | 13:57 |
| *** pcaruana has joined #zuul | 14:04 | |
| *** michael-beaver has joined #zuul | 14:04 | |
| *** jamesmcarthur has quit IRC | 14:09 | |
| bolg | tobiash: avass: there has been actually a change not so long ago, the sub-set was checked using a fast failing loop which should be interpretted as "<=" (see https://review.opendev.org/#/c/681936/6/zuul/driver/gerrit/gerritconnection.py) | 14:14 |
| *** avass has quit IRC | 14:16 | |
| Shrews | oh, recheck is a bot? i feel silly now | 14:17 |
| *** bolg_ has joined #zuul | 14:18 | |
| *** bolg has quit IRC | 14:19 | |
| Shrews | i wonder if fungi is a bot, too | 14:19 |
| *** recheck has quit IRC | 14:20 | |
| *** recheck has joined #zuul | 14:20 | |
| *** recheck has quit IRC | 14:20 | |
| *** recheck has joined #zuul | 14:21 | |
| *** recheck has quit IRC | 14:21 | |
| *** recheck has joined #zuul | 14:21 | |
| *** brendangalloway has quit IRC | 14:21 | |
| *** jangutter has joined #zuul | 14:25 | |
| *** electrofelix has joined #zuul | 14:25 | |
| Shrews | zbr: If we're voting, I prefer not having the bot. I'm sure there are likely less intrusive ways for folks to monitor stackoverflow if they so choose. | 14:25 |
| *** jamesmcarthur has joined #zuul | 14:27 | |
| *** recheck has quit IRC | 14:27 | |
| zbr | Shrews: done, removed. | 14:30 |
| Shrews | zbr: well, i was just voicing *my* preference. might want to check with others, too. | 14:30 |
| zbr | Shrews: two reports were enough, it started as an experiment, proved usedful in some cases but detrimental in others. | 14:31 |
| Shrews | *nod* | 14:31 |
| zbr | main issue is that stackoverflow feed does not have an option to filter only questions with votes, so a lot of very-low-quality questions can flood a channel. | 14:32 |
| zbr | you can only imagine what happened when I activated "pip" tag for pypa channel ;) | 14:32 |
| Shrews | i don't think many (if any) of us want to include stackoverflow in our normal support path. i could be mistaken though | 14:33 |
| *** bhavikdbavishi has joined #zuul | 14:33 | |
| fungi | web forums like stackoverflow could be viewed as social networking and outreach opportunities, but yeah, i hope nobody considers that an official means of getting support for free software | 14:36 |
| Shrews | though i could see some benefit to it, since that site is usually the top hit from search engines in tech-related questions | 14:36 |
| openstackgerrit | Merged zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 14:48 |
| Shrews | tristanC: do we need to add some sort of "wait" period in the k8s job? hit the "pod status is Pending" timeout again: https://zuul.opendev.org/t/zuul/build/7ca81f62f8ad488e824ae83e559154c1/log/job-output.txt#1152 | 14:49 |
| Shrews | not sure if that was also related to the ipv6 issues | 14:51 |
| clarkb | Shrews: that ran in ovh bhs1 which does not have ipv6 at all | 14:52 |
| clarkb | I doubt this is the problem we saw in fortnebula | 14:52 |
| Shrews | clarkb: oh? a public_ipv6 value is listed for the node | 14:54 |
| Shrews | https://d1a899ea13e66d0e8401-c80779e78a1a20b721125eb9333742c5.ssl.cf5.rackcdn.com/687965/2/gate/nodepool-functional-k8s/7ca81f6/zuul-info/inventory.yaml | 14:54 |
| Shrews | not sure what that is then | 14:54 |
| clarkb | Shrews: that is an ovh oddity, basically their neutron says "hey you can use this address" but then config drive doesn't include any of that info so we never configure it | 14:55 |
| clarkb | Shrews: this is why we force ipv4 in our clouds.yaml for ovh clouds | 14:55 |
| Shrews | ah, i see | 14:55 |
| clarkb | in theory we could have the zuul job statically configure that address if we wanted | 14:56 |
| clarkb | but I don't think it has come up as being a need so has mostly just been ignored | 14:56 |
| tristanC | Shrews: kubernetes driver is waiting 300 seconds for pod to become running | 14:57 |
| tristanC | and it seems like we don't collect k8s api logs, thus i don't know why the pod didn't transition to running | 14:59 |
| Shrews | we should probably collect those then | 15:01 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch https://review.opendev.org/644927 | 15:03 |
| zbr | clarkb: fungi: https://review.opendev.org/#/c/688617/9 (centos8) is also ready. | 15:07 |
| fungi | zbr: yep, i was still working my way through reviewing that one | 15:08 |
| * fungi is a slow, distracted, reader | 15:08 | |
| clarkb | Shrews: I'd like to restart nodepool launchers after my morning meeting. This should pick up the DELETED status handling for nova instances. Anything else I should keep in mind? | 15:09 |
| zbr | fungi: thanks. | 15:09 |
| openstackgerrit | Merged zuul/nodepool master: Fix builder shutdown race in tests https://review.opendev.org/687965 | 15:24 |
| *** bhavikdbavishi has quit IRC | 15:31 | |
| *** bolg_ has quit IRC | 15:32 | |
| *** igordc has joined #zuul | 15:37 | |
| *** mattw4 has joined #zuul | 15:38 | |
| *** jpena is now known as jpena|brb | 15:45 | |
| openstackgerrit | Merged zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 15:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 15:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 15:58 |
| *** themr0c has quit IRC | 16:02 | |
| *** jpena|brb is now known as jpena | 16:18 | |
| *** igordc has quit IRC | 16:31 | |
| *** hashar has quit IRC | 16:38 | |
| tristanC | corvus: the namespace and buildset test stack for zuul-registry, should i +3 the first change? | 16:39 |
| corvus | tristanC: yes, i think it's ready | 16:40 |
| corvus | weird -- minikube with cri-o still wants a "docker" executable for something? | 16:41 |
| corvus | https://zuul.opendev.org/t/zuul/build/acd2f48e57f349829c1fffb95ef8b1c3/console#1/0/16/ubuntu-bionic | 16:42 |
| corvus | my test vm did have docker installed, so i haven't seen that before | 16:42 |
| corvus | "Using this driver requires Docker and a Linux environment but not a hypervisor" | 16:44 |
| corvus | apparently if you use vm driver==none it uses docker for something | 16:44 |
| corvus | well, i think the 2 can coexist :) | 16:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 16:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 16:46 |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Fix github auth token authentication https://review.opendev.org/686355 | 16:47 |
| tristanC | corvus: perhaps the kube-apiserver is also started in a container and minikube rely on docker for that? | 16:48 |
| corvus | tristanC: yeah, i think that's the case. | 16:49 |
| corvus | it looks like it runs most of the stuff in docker containers except for kubelet | 16:50 |
| tristanC | corvus: hum, perhaps there is also an issue with the code according to: https://github.com/kubernetes/minikube/issues/5549 | 16:51 |
| corvus | tristanC: wow you're better at googling than i am :) | 16:52 |
| tristanC | corvus: to be sure we need to check that kubelet is started with "--container-runtime=remote" and "--container-runtime-endpoint unix://...crio.sock" | 16:53 |
| tristanC | corvus: also "crictl version" is a simple test to verify cri-o is started correctly | 16:54 |
| corvus | i'm not having trouble with k8s using crio :) | 16:54 |
| Shrews | clarkb: i'm not aware of anything | 16:54 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Mandatory SQL connection https://review.opendev.org/688755 | 16:59 |
| *** jpena is now known as jpena|off | 17:01 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Mandatory SQL connection https://review.opendev.org/688755 | 17:04 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: fix mirror_fqdn variable https://review.opendev.org/688468 | 17:19 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 17:22 |
| mordred | corvus: podman has a docker compat shim ... | 17:26 |
| mordred | so that you can type docker and it'll run podman | 17:26 |
| mordred | don't know if that's helpful for minikube or not | 17:26 |
| clarkb | you get that by default on rhel8 if you install docker too | 17:27 |
| clarkb | (this has apparently surprised some people) | 17:27 |
| mordred | yeah. I think from the ppa it's a separate package | 17:27 |
| mordred | hrm. doesn't look like it. I think it's actually just a symlink anyway | 17:28 |
| *** electrofelix has quit IRC | 17:32 | |
| *** spsurya has quit IRC | 17:33 | |
| corvus | mordred: oh cool -- well maybe we can try that later -- but now i'm hitting "Failed to enable container runtime: running command: sudo systemctl restart crio: exit status 1" | 17:34 |
| corvus | so i think i'm going to have to hold a node | 17:34 |
| tristanC | corvus: i spent quite sometime doing crio configuration, i may be able to help with that | 17:36 |
| corvus | tristanC: cool, thanks :) | 17:36 |
| corvus | (i do have this working on a test vm -- so now that this is running on a clean vm, i'm probably just missing some small step i either forgot i did, or accidentally did and didn't realize it) | 17:37 |
| corvus | at least, that's what i'm hoping | 17:37 |
| mordred | corvus: I'm going to blame systemd | 17:38 |
| corvus | it's not not involved | 17:38 |
| corvus | i think we should probably get some journal logs too | 17:38 |
| *** armstrongs has joined #zuul | 17:43 | |
| *** armstrongs has quit IRC | 17:53 | |
| *** hashar has joined #zuul | 18:01 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:05 |
| *** openstackgerrit has quit IRC | 18:07 | |
| *** jangutter has quit IRC | 18:08 | |
| *** jamesmcarthur has quit IRC | 18:12 | |
| corvus | Shrews: oh that's nice, i didn't know about that :) | 18:14 |
| corvus | Shrews: can you just run "/tmp/minikube logs > ~/logs/minikube.txt" ? | 18:15 |
| corvus | Shrews: i think that the base job will pull that automatically | 18:15 |
| Shrews | corvus: i didn't know about it either until just now when i went a' googlin'. maybe? i guess that would have to be using the shell module | 18:16 |
| corvus | ya | 18:16 |
| corvus | probably more efficient than storing it all in ansible's memory too | 18:16 |
| Shrews | corvus: i'll try that after the k8s job runs. just want to verify this will work first | 18:17 |
| corvus | ya | 18:17 |
| corvus | tristanC: ah yep, it was something i fixed on my test node and forgot about: http://paste.openstack.org/show/784003/ | 18:20 |
| corvus | tristanC: it looks like a bug in the debian packaging -- the config file is set up for /usr/libexec/crio/conmon but the installed path is /usr/bin/conmon and it doesn't create a link | 18:20 |
| corvus | tristanC: do you think they want bugs here? https://bugs.launchpad.net/~projectatomic | 18:21 |
| corvus | meanwhile, we can either sed the conf file or make a symlink... | 18:22 |
| *** chandankumar is now known as raukadah | 18:24 | |
| corvus | Shrews: i'll incorporate that into my next ps | 18:24 |
| mordred | corvus: I imagine they want bugs on their github | 18:24 |
| corvus | Shrews: (of the minikube crio test) | 18:24 |
| corvus | mordred: is there a github repo for the ppa? | 18:24 |
| mordred | corvus: https://github.com/containers/libpod/issues | 18:25 |
| *** openstackgerrit has joined #zuul | 18:25 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:25 |
| mordred | corvus: well - last time I talked with them about the ppa, I believe I did it on the libpod repo | 18:25 |
| corvus | oh, ok | 18:26 |
| Shrews | corvus: there is also a ~/.minikube/logs directory, but there is nothing in it when i test locally. not sure what that's about | 18:27 |
| mordred | corvus: oh - although that's the cri-o package, yeah? | 18:27 |
| mordred | corvus: so it might be in https://github.com/cri-o/cri-o | 18:27 |
| corvus | mordred: yeah | 18:27 |
| mordred | corvus: I'm honestly not 100% sure how that all hangs together | 18:27 |
| corvus | mordred: well, actually the 'conmon' package | 18:27 |
| mordred | corvus: probably libpod | 18:28 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 18:30 |
| Shrews | corvus: looks like we need to not use ~ there because of the 'become' part. just failed in my change | 18:34 |
| corvus | oh right | 18:34 |
| corvus | ansible_remote_user i guess? | 18:35 |
| corvus | and i guess we need become to get journal logs, etc? | 18:35 |
| *** igordc has joined #zuul | 18:36 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:36 |
| Shrews | corvus: ^^ | 18:37 |
| Shrews | ansible_user_dir | 18:37 |
| corvus | kk, i'll update my copy after the next failure :) | 18:37 |
| Shrews | corvus: we use 'become' when starting minikube which creates a permission issue. locally i got this error: | 18:39 |
| Shrews | $ ./minikube logs | 18:39 |
| Shrews | 💣 Error getting config: open /home/shrews/.minikube/profiles/minikube/config.json: permission denied | 18:39 |
| * Shrews wonders if the 'bomb' emoji came across | 18:39 | |
| mordred | Shrews: I see the bomb | 18:40 |
| Shrews | mordred: YOU'RE the bomb | 18:40 |
| corvus | yeah... there's like... multiple issues on github about how to configure your system to correctly see the error emojis... | 18:40 |
| * mordred headdesks | 18:40 | |
| clarkb | I think it largely comes down to font glyphs | 18:40 |
| corvus | i feel like... maybe that is a misdirection of energy.... | 18:40 |
| clarkb | I don't see that one but see others | 18:41 |
| Shrews | corvus: you NEED emojis | 18:41 |
| corvus | Shrews: if i can't type "sad-emoji" i can't ever really be sad | 18:41 |
| *** michael-beaver has quit IRC | 18:44 | |
| Shrews | corvus: other than the sans-docker part, what is the advantage of cri-o? | 18:48 |
| Shrews | oh, integration with our registry i guess | 18:49 |
| corvus | Shrews: the zuul-registry will be able to hold speculative images for more than one upstream registry (dockerhub, quay.io, gcr, etc) but that will only work with non-docker runtimes. so if we want to use non-dockerhub speculitave images with k8s, we'll need this. | 18:49 |
| AJaeger | Shrews: sans-docker is enough advantage for me ;) | 18:59 |
| *** jamesmcarthur has joined #zuul | 19:00 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:07 |
| *** hashar has quit IRC | 19:28 | |
| *** hashar has joined #zuul | 19:36 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:37 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:39 |
| mnaser | hmm | 19:42 |
| mnaser | so when moving gear to the opendev tenant | 19:42 |
| mnaser | i guess we missed the fact that we didnt actaully run any jobs here - https://review.opendev.org/#/c/688455/ | 19:42 |
| mnaser | im not sure why we didn't run any jobs there | 19:43 |
| mnaser | and the actual thing is broken now because bindep.txt in opendev is empty afaik | 19:43 |
| mnaser | so its needing something with ffi | 19:43 |
| clarkb | we ran jobs there... | 19:44 |
| mnaser | ah gah | 19:44 |
| mnaser | i got tricked by the promote jobs overriding that section again | 19:44 |
| clarkb | and the ffi problem should be fixed by cffi publishing wheels | 19:44 |
| Shrews | mnaser: that gets me all the time, too | 19:44 |
| mnaser | i wonder if i can fix that | 19:45 |
| mnaser | is the source code somewhere for it | 19:45 |
| mnaser | oh hmm | 19:46 |
| mnaser | https://review.opendev.org/#/c/688683/ -- doesnt seem to be broken in this for example | 19:46 |
| clarkb | we don't annotate the pipeline names outside of the openstack tenant maybe? | 19:47 |
| clarkb | we can update that if that is the problem | 19:47 |
| mordred | checks plugin will make all of that happy :) | 19:47 |
| fungi | i've also been wondering whether we should report the tenant name as well as the pipeline, now that we potentially have multiple tenants reporting separate buildsets with the same pipeline names | 19:48 |
| corvus | that should never happen | 19:48 |
| mnaser | i wonder if its because success-message is not defined for opendev | 19:49 |
| mnaser | and maybe that thing makes introspection to find out which pipeline it ran in> | 19:49 |
| mnaser | thing = the piece of js code we have in gerrit | 19:49 |
| corvus | ie, a project should only be configured for reporting pipelines in one tenant | 19:49 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:50 |
| mnaser | https://review.opendev.org/static/hideci.js?e=054a8d9d1a4a2e49b7411eb93fd173a7 --- var pipelineNameRegex = /Build \w+ \(([-\w]+) pipeline\)/; | 19:50 |
| mnaser | i can push up a quick patch to change that (unless the checks plugin is going live next week, mordred :p) | 19:51 |
| clarkb | mnaser: it won't be you should push up the pipeline update | 19:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 19:51 |
| fungi | corvus: on a related note, did you see the example earlier today where the opendev zuul triple-reported a merge failure? (tristanC's theory is that's because it's included in three tenants) | 19:52 |
| mordred | yeah. checks plugin is still in dev - mostly just saying it'll be so nice when there's a non-scraping way of doing this properly | 19:52 |
| * mnaser looks at that javascript | 19:52 | |
| mnaser | Copyright (c) 2014 Hewlett-Packard Development Company, L.P. | 19:52 |
| corvus | fungi: nope, someone should look into that :) | 19:53 |
| fungi | corvus: tobiash mentioned it at 13:43z... https://review.opendev.org/659991 | 19:53 |
| mordred | mnaser: is a real big hack that we're very unpleased about the existence of - thus the excitement about checks :) | 19:53 |
| corvus | fungi: it should only do that for pipelines it's part of. note, however, that the openstack tenant in openstack adds *every* project to the check pipeline. | 19:55 |
| mnaser | btw -- looks like i got the gear image to build on the first try -- but no functional tests .. so i dont know how we want to proceed with reviewing something like this: https://review.opendev.org/#/c/688446/ | 19:57 |
| Shrews | corvus: fyi, i'm uncertain if all of the environment vars in https://review.opendev.org/#/c/688775/7/playbooks/nodepool-functional-k8s/post.yaml are needed, but you'll need at least the MINIKUBE_HOME one on your change I'm guessing. I planned to try just that one after a successful log pull. | 19:58 |
| Shrews | which looks like just happened | 19:58 |
| mordred | mnaser, corvus: we _could_ get fancy and make the zuul-quickstart run gear as a separate container using that image instead of the internal one - just so we have a good functional cross-test for the image | 19:58 |
| Shrews | corvus: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_194/688775/7/check/nodepool-functional-k8s/194d14d/ubuntu-xenial/minikube.txt | 19:58 |
| mordred | we don't _need_ that to be split out in the quickstart though - so it might be counterproductive | 19:58 |
| mnaser | mordred: yeah, i mean it probably doesnt hurt that much more for quickstart to having it externalized | 19:59 |
| mnaser | "what's one more container": | 20:00 |
| mordred | mnaser: yah - my main concern would be making it seem more complex for a user learing - but maybe having an explicit server helps see how the system hangs together? | 20:00 |
| Shrews | tristanC: does https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_194/688775/7/check/nodepool-functional-k8s/194d14d/ubuntu-xenial/minikube.txt look like it would have the logging info you would need to diagnose the k8s failure? | 20:00 |
| corvus | mnaser, mordred: i'd say let's not. quickstart is supposed to be mostly educational... and running geard separately isn't how we recommend doing it | 20:01 |
| ianw | corvus: when is the next zuul release likely? | 20:01 |
| mnaser | i mean i'm happy to do my own testing and iterate on it, yes this might mean we release broken images but.. | 20:01 |
| corvus | mnaser: should be easy to throw a "run a job" test at it at least | 20:02 |
| corvus | ianw: i guess now or tomorrow? :) | 20:03 |
| corvus | Shrews: is nodepool releasable? | 20:03 |
| mnaser | ok i guess i can try and use that image to spin up a container with geard and then use the container to try and enqueue a job or something | 20:03 |
| corvus | mnaser: i'm also personally okay with publishing that image as-is. at least until the first patch comes in that says "fix issue with docker image..." :) | 20:04 |
| tristanC | Shrews: seems like it yes | 20:04 |
| Shrews | corvus: i believe so, though we only just restarted the launchers with the latest changes today | 20:04 |
| fungi | if there are zuul/nodepool releases today i'll link to the corresponding release announcements in the zuul section of the osf newsletter | 20:05 |
| mordred | mnaser: same here | 20:05 |
| corvus | Shrews, ianw: i think we should release zuul+nodepool together because of the auto python version thingy | 20:05 |
| Shrews | tristanC: awesome | 20:05 |
| tristanC | Shrews: though i don't know how minikube works, perhaps we also needs the output of the kube-system pods. | 20:05 |
| ianw | corvus: yeah, that was my thought -- but the nodepool side isn't in. we were going to wait for a zuul release, then reference it in the nodepool change which would go into next nodepool release | 20:05 |
| Shrews | tristanC: do you know how to get that? i don't think any of us know minikube | 20:05 |
| corvus | ianw: oh ha, ok | 20:05 |
| tristanC | Shrews: well it seems like they are dumped too | 20:06 |
| ianw | i mean that was why i was asking, to see where that chnage was. was going to mention it wrt centos8 (python3 only) | 20:06 |
| mnaser | mordred, corvus: i can try to consume it once we publish it.. i'm going to be honest, i find it a little harder for me to get tests up cause im not sure if i can say.. run a "gearman tempest" against it or something :-P | 20:06 |
| corvus | ianw: it seems that we did the opposite: https://zuul-ci.org/docs/zuul/releasenotes.html | 20:06 |
| ianw | hrm, yeah that probably shouldn't have been a releasenote in hindsight | 20:07 |
| clarkb | I status logged the launcher version we started on seems to be working no objection to tagging that from me | 20:09 |
| *** jamesmcarthur has quit IRC | 20:09 | |
| mordred | I have no objections to tagging anything | 20:09 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 20:10 |
| corvus | ianw: okay after reviewing the discussions, i think "zuul then nodepool" makes sense. do you want to release zuul with that release note, or alter or remove it first? | 20:10 |
| ianw | corvus: just working on a removal, to avoid confusion | 20:10 |
| corvus | (we could leave it as is, or alter it to say "in an upcoming nodepool release...", or remove it) | 20:10 |
| corvus | ianw: cool, then the plan is to wait for the release note change, then release zuul, then merge the nodepool change and release it (probably 1 day later?) | 20:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Remove python-path auto release note https://review.opendev.org/688809 | 20:12 |
| ianw | corvus: right, no urgent rush. can update the nodepool change to say "this relies on zuul <release>" | 20:13 |
| *** jamesmcarthur has joined #zuul | 20:19 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 20:21 |
| Shrews | corvus: fyi, MINIKUBE_HOME: "{{ ansible_user_dir }}" is all you need | 20:22 |
| corvus | Shrews: cool, thanks! | 20:24 |
| *** pcaruana has quit IRC | 20:25 | |
| *** pcaruana has joined #zuul | 20:34 | |
| mnaser | so uh | 20:36 |
| mnaser | working with some of the rdo folks on debugging an issue with the tripleo-reproducer stuff (cc nhicher !) and i noticed the following | 20:36 |
| mnaser | https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/callback/zuul_stream.py#L140-L144 | 20:36 |
| mnaser | https://bugs.launchpad.net/tripleo/+bug/1833465 -- seems like they run int othis often, and the generic exception handler there kinda masks possible issue | 20:36 |
| openstack | Launchpad bug 1833465 in tripleo "tripleo reproducer fails w/ "waiting on logger"" [Critical,Triaged] | 20:36 |
| mnaser | does anyone recall a historical reason why we dont log the actual exception for example or how we I rewrite that to help uncover the actual issues | 20:37 |
| mnaser | interestingly enough it always happens after iptables-save and then never recovers | 20:38 |
| corvus | mnaser: the job content probably blocked port 19885 | 20:38 |
| *** jamesmcarthur has quit IRC | 20:38 | |
| mnaser | corvus: that was my guess but also this is a thing that doesnt consistently fail | 20:39 |
| mnaser | so its an intermittant once-in-a-while-but-sometimes-kinda-consistent | 20:39 |
| corvus | mnaser: i don't think having the actual exception is going to help -- it's just going to say "connection refused" or similar. either way, the error is "can not open network connection to port 19885" on node | 20:39 |
| mnaser | iptables shows that it's happily accepted packets for a while too, i was trying to see if i can find the log of zuul_console without luck | 20:40 |
| corvus | mnaser: by "log of zuul_console" what do you mean? | 20:41 |
| mnaser | corvus: the actual thing that listens on port 19885 | 20:42 |
| corvus | mnaser: it doesn't have its own log files; it may (in rare cases) return an error to ansible | 20:43 |
| corvus | mnaser: i think the way to track that down is to hold a node and inspect connectivity to 19885 | 20:44 |
| mordred | ooh - maybe I can nerd-snipe mnaser into finishing my half-done patches to rework log streaming ... | 20:46 |
| mordred | although maybe I shouldn't say that out loud like that ... | 20:46 |
| mnaser | corvus: in this case, inspecting the held node, i dont see any issues. nothing is running at that port anymore, but i can start up SimpleHTTPServer at that port and it responds.. | 20:47 |
| corvus | mnaser: did the node reboot? | 20:47 |
| mnaser | mordred: dont threaten me with a good time :p | 20:47 |
| mnaser | corvus: it did not, this would happen after persisting iptables (and i have the journal logs here showing that ansible actually continued to run with no problems) | 20:48 |
| mnaser | but the logs died | 20:48 |
| *** jamesmcarthur has joined #zuul | 20:48 | |
| corvus | mnaser: anything in syslog/journal about it? oom? | 20:48 |
| corvus | systemd may have recorded reaping the process | 20:48 |
| mnaser | nothing sticks out in journal, no ooms, though i'm trying to grep for 'stream' or 'console' and not seeing much about it there (does it use ansible to start it up?) | 20:49 |
| corvus | mnaser: yes, it's an ansible module that daemonizes | 20:49 |
| mnaser | that's what i remember too, but maybe because it daemonizes then ansible never logs it in journal like it usually does | 20:50 |
| *** pcaruana has quit IRC | 20:51 | |
| mnaser | earliest record of ansible doing things is clearly a bunch of the ssh setup | 20:52 |
| mnaser | interstingly enough | 20:53 |
| mnaser | zuul_console.json has all the right things | 20:53 |
| mnaser | which means it didnt die | 20:54 |
| corvus | mnaser: that's logged by the callback handler and doesn't rely on the log streamer | 20:54 |
| Shrews | folks, let's merge https://review.opendev.org/688775 today if we can, please. random fails keep biting us | 21:00 |
| mordred | mnaser: https://review.opendev.org/#/q/topic:zuul-stream-rework if you find yourself with time and feel like getting nerd-sniped. it's still on my TDL - but it's buried a bit below other things | 21:00 |
| mnaser | mordred: oou interesting and doesnt even seem that complex.. initially :L | 21:01 |
| mordred | mnaser: yeah - the first 80% of that one is TOTALLY straightforward | 21:02 |
| corvus | Shrews: are you sure we can't skip the synchronize step if we put it in zuul-output/logs ? | 21:02 |
| mordred | corvus: I was going to ask the same question - but the rest of the playbook is also not using zuul-output ... | 21:02 |
| Shrews | corvus: we could, but that differs from how we do the other logs | 21:03 |
| corvus | have to start somewhere | 21:03 |
| mordred | corvus: so maybe we should do a patch wich updates it to change the whole playbook | 21:03 |
| corvus | the other logs are doing something else | 21:03 |
| Shrews | why do our docs recommend ~/work/logs then? | 21:03 |
| corvus | Shrews: the difference is between whether it happens on the executor or the remote host | 21:03 |
| corvus | this case is "run a command on the remote host to get a log file" that's exactly the use-case that zuul-output is designed for | 21:04 |
| corvus | i think the best way to update the other logfiles would be to have nodepool write them directly into ~zuul/zuul-output/logs | 21:04 |
| corvus | that's why i say it's a different thing | 21:04 |
| corvus | the zuul-output thing is *supposed* to make this simple :) | 21:05 |
| corvus | but we have to actually start using it at some point if we want that to happen | 21:05 |
| corvus | Shrews: if you just want to merge that, i'm fine -- i'll do it the way i'm suggesting in the kubernetes role | 21:05 |
| corvus | i just think i was misunderstood in my review of the change, and i think it's important to straigten this out | 21:06 |
| Shrews | i wasn't even aware of that being an option (or, at least, i forgot it was?) | 21:06 |
| Shrews | corvus: let's merge as is, then i'll change all of the logging in one chunk in another review | 21:07 |
| corvus | Shrews: the idea is to move everything in zuul-jobs to use this system | 21:07 |
| corvus | but it's still only half-implemented and sparsely documented :/ | 21:08 |
| Shrews | is it implemented in a base job? | 21:08 |
| corvus | several folks have worked on it piecemeal | 21:08 |
| mordred | Shrews: yes | 21:08 |
| Shrews | well, i missed that. my fault, i guess | 21:08 |
| tristanC | btw there is topic:zuul-jobs-with-kubectl that is waiting for review to fix the fetch-* roles to use the zuul-output | 21:08 |
| corvus | Shrews: https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-fetch-output is the main thing | 21:08 |
| mordred | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/fetch-output | 21:09 |
| mordred | corvus link is better than mine | 21:09 |
| tristanC | (without topic:zuul-jobs-with-kubectl, most zuul-jobs doesn't work on kubernetes) | 21:10 |
| * mordred has that on his queue now | 21:10 | |
| corvus | tristanC: should we document that as a site var and name it "zuul_site_use_fetch_output" (eg like https://zuul-ci.org/docs/zuul-jobs/mirror.html )? | 21:11 |
| corvus | tristanC: alternatively -- we could just let folks know that zuul-jobs is going to rely on fetch-output and not make it an option | 21:12 |
| mordred | we should add a zuul_site_use_fetch_output to opendev's site vars, since we have fetch-output | 21:12 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 21:12 |
| mordred | corvus: could we use zuul_return in fetch-output to set zuul_use_fetch_output to true? | 21:12 |
| Shrews | ok ^^. i'll fix up the nodepool portion tomorrow | 21:12 |
| mordred | corvus: wait- that's the dumbest thing I've said | 21:13 |
| corvus | mordred: no, only affects later playbooks | 21:13 |
| mordred | I blame jetlag | 21:13 |
| tristanC | corvus: iiuc, site vars are not tenant scoped, and that makes it difficult to use them base job that may be different accross tenants | 21:13 |
| corvus | (well, we could try some set_fact or something) | 21:13 |
| Shrews | fwiw, i read "Zuul executor creates a directory to hold all of the content related to the job" in the docs as "on the node". maybe i can clarify that tomorrow too | 21:13 |
| corvus | tristanC: you can define them in base jobs | 21:13 |
| tristanC | for* base job | 21:13 |
| corvus | Shrews: fwiw, the zuul executor does *nothing* on any node | 21:14 |
| corvus | tristanC: so if you have a situation where one tenant uses fetch-output and one doesn't, you could just set the variable there | 21:14 |
| corvus | tristanC, mordred: but i really like the idea of saying "zuul-jobs depends on fetch-output" | 21:15 |
| mordred | corvus: I thnik I like te variable approach like tristanC has in these patches - it's already implemented and it removes a potential downstream breaking point... | 21:15 |
| mordred | THAT SAID ... | 21:15 |
| mordred | I also want to say zuul-jobs depends on fetch-output | 21:15 |
| corvus | every role/job in zuul-jobs is going to need that kind of branching logic | 21:15 |
| corvus | that's too much to ask | 21:15 |
| mordred | indeed. maybe we should go ahead and send out the warning email to the mailing list "please make sure your base jobs have fetch-output in them" | 21:17 |
| tristanC | mordred: corvus: either way works for me, though the toggle makes it backward compatible for existing user | 21:17 |
| tristanC | i'd be happy to rework the stack to remove non zuul-output support if you prefer that | 21:17 |
| mordred | but go ahead and land the ones tristan has here to keep moving the needle forward (and fix those roles for k8s) while we wait for the appropriate time to pass for breaking changes | 21:17 |
| mordred | then remove the branching as a cleanup? | 21:18 |
| mordred | tristanC: software factory is using fetch-output already yeah? | 21:19 |
| corvus | if it's not urgent, i'd prefer to avoid landing the branching at all... because the cleanup will be a lot of work... | 21:19 |
| mordred | SpamapS: about about your zuul? do you use fetch-output in your base job? | 21:19 |
| mordred | pabelanger: ^^ ? | 21:19 |
| corvus | but if tristanC is motivated enough to not only land this series, but also write and get the cleanups landed, i won't object... | 21:19 |
| *** hashar has quit IRC | 21:20 | |
| corvus | (i'm mostly trying to reduce everyone's workload :) | 21:20 |
| mordred | ++ | 21:20 |
| * mordred is happy to go either route | 21:20 | |
| mordred | I suppose either route involves another set of patches for those roles from tristanC :) | 21:20 |
| tristanC | mordred: not yet, we are waiting for that stack to land. | 21:20 |
| mordred | tristanC: gotcha | 21:20 |
| tristanC | aa/clear | 21:37 |
| tristanC | oops | 21:37 |
| *** jamesmcarthur has quit IRC | 21:45 | |
| SpamapS | mordred: maybe? What's fetch-output exactly? | 21:47 |
| SpamapS | doesn't look like it actually | 21:48 |
| mordred | SpamapS: role to be used in base jobs that fetches stuff back from the node(s) to the executor | 21:48 |
| mordred | SpamapS: https://zuul-ci.org/docs/zuul-jobs/log-roles.html#role-fetch-output | 21:49 |
| mordred | SpamapS: the general goal has been to get everyone using fetch-output so that indivual jobs don't need to know how to handle differences between k8s and vms for instance | 21:49 |
| SpamapS | Oh, I use zuul-base-jobs so it probably calls that | 21:49 |
| corvus | yep | 21:50 |
| corvus | zuul-base-jobs has both ensure-output-dirs and fetch-output | 21:50 |
| mordred | woot | 21:51 |
| SpamapS | as much grief as using that has caused me, I suspect it has saved me a lot of work to just use it via the git driver. :) | 21:51 |
| fungi | SpamapS: yep, https://opendev.org/zuul/zuul-base-jobs/src/branch/master/playbooks/base/post.yaml#L3 | 21:54 |
| clarkb | mnaser: fwiw the last time I debugged similar issues in tripleo jobs I discovered the test VM was crashing due to use of nested virt. Sounds like you've already ruled out the possibility of a reboot, but I wanted to call it out because no one had noticed it happenign until I looked at their syslog logs | 22:00 |
| mnaser | clarkb: yeah it doesnt even get that far, its in the prep phase, but still kinda trying to grok whats going on | 22:01 |
| mordred | Shrews: you're missing a logs dir in there | 22:03 |
| mordred | Shrews: and in saying that - I realize that the docs for fetch-output do not actually document the subdirs | 22:04 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 22:05 |
| *** jamesmcarthur has joined #zuul | 22:13 | |
| *** jamesmcarthur has quit IRC | 22:31 | |
| corvus | Shrews: working code in https://review.opendev.org/688578 produces https://zuul.opendev.org/t/zuul/build/9f700416bd00475db3be1110d0f5ca20/log/minikube.txt | 22:32 |
| corvus | tristanC: crio and minikube seem to disagree on which cgroup driver to use -- crio.conf has 'systemd' and minikube configured kubelet with --cgroup-driver=cgroupfs. do you have an opinion on which we should use? | 22:36 |
| corvus | (presumably we can either sed crio.conf or add another argument to minikube) | 22:36 |
| corvus | (i believe in my testing i changed crio to use cgroupfs) | 22:37 |
| tristanC | corvus: you can find my handmade crio.conf conf here: https://github.com/podenv/silverkube/blob/master/silverkube.py#L216 | 22:41 |
| corvus | tristanC: cool, i'll edit the conf then :) | 22:41 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 22:43 |
| tristanC | corvus: on fedora, systemd was also missing some accounting options, you might need to add that too: https://github.com/podenv/silverkube/blob/master/silverkube.py#L683 | 22:44 |
| *** mattw4 has quit IRC | 22:45 | |
| *** jamesmcarthur has joined #zuul | 23:35 | |
| *** tosky has quit IRC | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:59 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!