| *** jamesmcarthur has quit IRC | 00:14 | |
| *** jamesmcarthur has joined #openstack-infra | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:20 | |
| *** jamesmcarthur has joined #openstack-infra | 00:39 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Fix zuul-stream-functional tests https://review.opendev.org/694619 | 00:41 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 00:45 | |
| *** tosky has quit IRC | 00:46 | |
| *** jamesmcarthur has joined #openstack-infra | 00:52 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** diablo_rojo__ has quit IRC | 01:01 | |
| *** jamesmcarthur has joined #openstack-infra | 01:14 | |
| *** jamesmcarthur has quit IRC | 01:14 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** ociuhandu has joined #openstack-infra | 01:31 | |
| *** jamesmcarthur has quit IRC | 01:33 | |
| *** ociuhandu has quit IRC | 01:41 | |
| *** goldyfruit_ has quit IRC | 01:53 | |
| *** jamesmcarthur has joined #openstack-infra | 02:17 | |
| *** ociuhandu has joined #openstack-infra | 02:26 | |
| *** ociuhandu has quit IRC | 02:30 | |
| *** jamesmcarthur has quit IRC | 02:36 | |
| *** ramishra has joined #openstack-infra | 02:39 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add user to shadow file too https://review.opendev.org/693647 | 02:51 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: create APP_DIR https://review.opendev.org/693646 | 02:51 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: docker: Base nodepool images on diskimage-builder base image https://review.opendev.org/694177 | 02:51 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 02:51 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile : install sudo for nodepool-builder https://review.opendev.org/694709 | 02:51 |
| *** ociuhandu has joined #openstack-infra | 03:00 | |
| *** ociuhandu has quit IRC | 03:07 | |
| *** ykarel has joined #openstack-infra | 03:15 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 03:49 |
| *** udesale has joined #openstack-infra | 04:02 | |
| *** ociuhandu has joined #openstack-infra | 04:27 | |
| *** ociuhandu has quit IRC | 04:29 | |
| *** ociuhandu has joined #openstack-infra | 04:30 | |
| *** ociuhandu has quit IRC | 04:36 | |
| *** ociuhandu has joined #openstack-infra | 04:53 | |
| *** ociuhandu has quit IRC | 04:58 | |
| *** ociuhandu has joined #openstack-infra | 04:58 | |
| *** ociuhandu has quit IRC | 05:03 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| ianw | mordred: so i'm a bit stuck on why https://review.opendev.org/693464 is not using the buildset registry for the images | 05:31 |
| *** ociuhandu has quit IRC | 05:35 | |
| ianw | here it is setting up the buildset registry -> https://zuul.opendev.org/t/zuul/build/85aff5d08e524985b99e6369018b4d0c/log/job-output.txt#495 | 05:35 |
| ianw | here is the pulls -> https://zuul.opendev.org/t/zuul/build/85aff5d08e524985b99e6369018b4d0c/log/job-output.txt#33498 | 05:36 |
| ianw | both of those are tags from the upstream images | 05:37 |
| ianw | hrm, maybe it needs a :latest? | 05:44 |
| *** raukadah is now known as chandankumar | 05:44 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 05:45 |
| *** slaweq has joined #openstack-infra | 05:48 | |
| *** slaweq has quit IRC | 05:57 | |
| *** kopecmartin has joined #openstack-infra | 05:57 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 06:12 |
| *** dychen has quit IRC | 06:16 | |
| ianw | afaict, you can't actually set an alternative default registry for docker? (https://github.com/moby/moby/issues/33069) | 06:17 |
| ianw | i think i might have to use "image: zuul-jobs.buildset-registry:5000/zuul/nodepool-builder" | 06:17 |
| ianw | which is fine, but i don't see other jobs doing that ... | 06:17 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 06:18 |
| *** jtomasek has joined #openstack-infra | 06:26 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: docker: Base nodepool images on diskimage-builder base image https://review.opendev.org/694177 | 06:31 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile : install sudo for nodepool-builder https://review.opendev.org/694709 | 06:31 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 06:32 |
| *** yolanda has quit IRC | 06:45 | |
| *** ykarel is now known as ykarel|afk | 06:46 | |
| *** dychen has joined #openstack-infra | 06:53 | |
| *** ykarel|afk is now known as ykarel|lunch | 06:55 | |
| *** ociuhandu has joined #openstack-infra | 07:02 | |
| *** rcernin has quit IRC | 07:04 | |
| *** ociuhandu has quit IRC | 07:06 | |
| *** rpioso is now known as rpioso|afk | 07:13 | |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| *** apetrich has joined #openstack-infra | 07:34 | |
| *** dpawlik has joined #openstack-infra | 07:34 | |
| *** slaweq has joined #openstack-infra | 07:38 | |
| *** jamesmcarthur has joined #openstack-infra | 07:39 | |
| *** pgaxatte has joined #openstack-infra | 07:49 | |
| *** igordc has joined #openstack-infra | 07:58 | |
| *** tkajinam has quit IRC | 08:02 | |
| *** igordc has quit IRC | 08:02 | |
| *** tesseract has joined #openstack-infra | 08:15 | |
| *** tosky has joined #openstack-infra | 08:20 | |
| *** jamesmcarthur has quit IRC | 08:21 | |
| *** ociuhandu has joined #openstack-infra | 08:22 | |
| *** jamesmcarthur has joined #openstack-infra | 08:23 | |
| *** dychen has quit IRC | 08:25 | |
| *** jamesmcarthur has quit IRC | 08:28 | |
| *** ociuhandu has quit IRC | 08:29 | |
| *** jamesmcarthur has joined #openstack-infra | 08:38 | |
| *** Lucas_Gray has joined #openstack-infra | 08:42 | |
| *** lucasagomes has joined #openstack-infra | 08:45 | |
| *** pkopec has joined #openstack-infra | 08:47 | |
| *** jaosorior has joined #openstack-infra | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** gfidente has joined #openstack-infra | 08:49 | |
| *** ralonsoh has joined #openstack-infra | 08:49 | |
| ianw | yay, "image:zuul-jobs.buildset-registry:5000" works! it's bringing in the latest images; and dib is failing in new ways | 08:50 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Update bindep.txt for some missing dependencies https://review.opendev.org/693975 | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Dockerfile: install vhd-util https://review.opendev.org/693976 | 08:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Start a buildset registry with docker image creation https://review.opendev.org/694175 | 08:52 |
| *** Vadmacs has joined #openstack-infra | 08:57 | |
| *** xek has joined #openstack-infra | 08:59 | |
| *** ociuhandu has joined #openstack-infra | 09:04 | |
| *** ykarel|lunch is now known as ykarel | 09:06 | |
| *** jamesmcarthur has quit IRC | 09:07 | |
| *** soniya29 has joined #openstack-infra | 09:09 | |
| *** jamesmcarthur has joined #openstack-infra | 09:12 | |
| *** jamesmcarthur has quit IRC | 09:17 | |
| *** yolanda has joined #openstack-infra | 09:22 | |
| *** jamesmcarthur has joined #openstack-infra | 09:38 | |
| *** iurygregory has joined #openstack-infra | 09:41 | |
| *** jamesmcarthur has quit IRC | 09:42 | |
| *** ociuhandu has quit IRC | 09:57 | |
| *** ociuhandu has joined #openstack-infra | 09:58 | |
| *** ociuhandu has quit IRC | 10:03 | |
| *** kaisers has quit IRC | 10:04 | |
| *** lennyb has quit IRC | 10:06 | |
| *** pcaruana has joined #openstack-infra | 10:10 | |
| *** jamesmcarthur has joined #openstack-infra | 10:12 | |
| *** owalsh has quit IRC | 10:12 | |
| *** jamesmcarthur has quit IRC | 10:17 | |
| *** kaisers has joined #openstack-infra | 10:20 | |
| *** sshnaidm|off is now known as sshnaidm|ruck | 10:20 | |
| *** owalsh has joined #openstack-infra | 10:22 | |
| *** ykarel_ has joined #openstack-infra | 10:28 | |
| *** ykarel has quit IRC | 10:30 | |
| *** kashyap has joined #openstack-infra | 10:36 | |
| kashyap | Hey, folks. Does the CI use Debian 'sid' at all? | 10:37 |
| *** rcernin has joined #openstack-infra | 10:52 | |
| AJaeger | kashyap: only buster and stretch, see https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L42-L44 | 10:52 |
| kashyap | AJaeger: Okay, thanks. Buster "stable", yeah? | 10:53 |
| AJaeger | I expect so but that's not my area of expertise | 10:54 |
| *** ykarel_ is now known as ykarel | 10:56 | |
| *** jamesmcarthur has joined #openstack-infra | 10:59 | |
| *** ociuhandu has joined #openstack-infra | 11:02 | |
| *** carl_cai has joined #openstack-infra | 11:03 | |
| *** jamesmcarthur has quit IRC | 11:04 | |
| *** pgaxatte has quit IRC | 11:07 | |
| *** ramishra has quit IRC | 11:14 | |
| *** udesale has quit IRC | 11:17 | |
| *** rfolco_pto has joined #openstack-infra | 11:50 | |
| *** dpawlik has quit IRC | 11:58 | |
| *** rfolco_pto is now known as rfolco | 11:58 | |
| *** jamesmcarthur has joined #openstack-infra | 12:00 | |
| *** dpawlik has joined #openstack-infra | 12:01 | |
| fungi | kashyap: yes. we've discussed adding unstable (sid) but someone would need to do the work of making sure the debian-minimal element in dib can produce images for it, and then adding package mirroring | 12:02 |
| fungi | we have stable (buster) and oldstable (stretch) images currently | 12:03 |
| kashyap | fungi: I see. I don't really need it; was just merely curious, as I was updating a page that tracks libvirt/QEMU/libguestfs versions across distros | 12:04 |
| kashyap | Tracking 'stable' is more than fine | 12:05 |
| *** roman_g has joined #openstack-infra | 12:05 | |
| fungi | it may be easier once we finish the work to switch dib over to bootstrapping from docker base images too | 12:05 |
| *** jamesmcarthur has quit IRC | 12:05 | |
| kashyap | What is 'dib'? | 12:06 |
| kashyap | Ah, Disk-Image-Builder | 12:07 |
| * kashyap --> lunch | 12:07 | |
| *** ociuhandu has quit IRC | 12:13 | |
| *** ykarel is now known as ykarel|afk | 12:14 | |
| *** jpena is now known as jpena|lunch | 12:26 | |
| *** ramishra has joined #openstack-infra | 12:26 | |
| *** rcernin has quit IRC | 12:31 | |
| *** dchen has quit IRC | 12:36 | |
| mordred | fungi, kashyap: yes - I'm pretty sure it will be | 12:41 |
| *** lpetrut has joined #openstack-infra | 12:42 | |
| *** ociuhandu has joined #openstack-infra | 12:43 | |
| mordred | fungi: I woke up at a reasonable time - how about you? | 12:43 |
| fungi | better, but not great. the nor'easter knocked out power out for a bit in the middle of the night and so with the flurry of beeping upses all over the house, but i think i mostly slept a solid 7 hours aside from that and so didn't awake for good until ~4am | 12:44 |
| mordred | fungi: ugh nor'easters | 12:47 |
| mordred | fungi: but - 4am is progress! | 12:47 |
| mordred | fungi: I napped for a couple of hours late afternoon - but then managed to stay up until midnight and sleep until 6am! | 12:49 |
| mordred | or "7am" as you call it on the east coast | 12:49 |
| *** rlandy has joined #openstack-infra | 12:56 | |
| *** rlandy is now known as rlandy|rover | 12:56 | |
| *** jamesmcarthur has joined #openstack-infra | 13:02 | |
| *** Lucas_Gray has quit IRC | 13:02 | |
| *** ociuhandu has quit IRC | 13:04 | |
| *** jamesmcarthur has quit IRC | 13:08 | |
| *** jamesmcarthur_ has joined #openstack-infra | 13:08 | |
| *** lennyb has joined #openstack-infra | 13:09 | |
| *** rh-jelabarre has joined #openstack-infra | 13:09 | |
| *** dtantsur is now known as dtantsur|bbl | 13:21 | |
| *** jpena|lunch is now known as jpena | 13:25 | |
| *** pgaxatte has joined #openstack-infra | 13:28 | |
| *** kjackal has joined #openstack-infra | 13:32 | |
| *** jamesmcarthur_ has quit IRC | 13:33 | |
| *** carl_cai has quit IRC | 13:33 | |
| *** weshay|ruck is now known as weshay | 13:40 | |
| *** KeithMnemonic has joined #openstack-infra | 13:40 | |
| *** rosmaita has joined #openstack-infra | 13:42 | |
| *** weshay has quit IRC | 13:46 | |
| *** whoami-rajat has joined #openstack-infra | 13:47 | |
| *** jamesmcarthur has joined #openstack-infra | 13:50 | |
| *** mriedem has joined #openstack-infra | 13:52 | |
| *** hwoarang has quit IRC | 13:52 | |
| *** Goneri has joined #openstack-infra | 13:57 | |
| *** hwoarang has joined #openstack-infra | 13:58 | |
| *** ramishra has quit IRC | 13:59 | |
| *** slaweq has quit IRC | 14:02 | |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: [WIP] Add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694776 | 14:02 |
| *** slaweq has joined #openstack-infra | 14:04 | |
| *** ramishra has joined #openstack-infra | 14:13 | |
| *** lennyb has quit IRC | 14:14 | |
| *** pgaxatte has quit IRC | 14:16 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 14:20 | |
| *** soniya29 has quit IRC | 14:21 | |
| *** Lucas_Gray has joined #openstack-infra | 14:23 | |
| *** Lucas_Gray has quit IRC | 14:25 | |
| *** Lucas_Gray has joined #openstack-infra | 14:28 | |
| *** beekneemech is now known as bnemec | 14:29 | |
| *** pgaxatte has joined #openstack-infra | 14:37 | |
| *** goldyfruit has joined #openstack-infra | 14:40 | |
| *** goldyfruit_ has joined #openstack-infra | 14:51 | |
| *** jamesmcarthur has quit IRC | 14:53 | |
| *** goldyfruit has quit IRC | 14:53 | |
| *** eharney has joined #openstack-infra | 14:57 | |
| *** rlandy|rover|mtg is now known as rlandy|rover | 15:00 | |
| *** tesseract has quit IRC | 15:01 | |
| *** tesseract has joined #openstack-infra | 15:01 | |
| *** dtantsur|bbl is now known as dtantsur | 15:01 | |
| *** rpioso|afk is now known as rpioso | 15:02 | |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: [WIP] Add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694776 | 15:05 |
| *** jamesmcarthur has joined #openstack-infra | 15:05 | |
| *** jamesmcarthur has quit IRC | 15:08 | |
| *** jamesmcarthur has joined #openstack-infra | 15:08 | |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: [WIP] Add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694776 | 15:08 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Change issue tracking for airshipctl https://review.opendev.org/694791 | 15:12 |
| *** dpawlik has quit IRC | 15:14 | |
| *** udesale has joined #openstack-infra | 15:22 | |
| *** ociuhandu has joined #openstack-infra | 15:26 | |
| *** ociuhandu has quit IRC | 15:28 | |
| clarkb | my sleep pattern has gotten betterbut still sick and discovered our water main was leaking 4' underground and now everything hurts because I am not built to operate ashovel | 15:53 |
| mordred | clarkb: GROSS | 15:55 |
| *** dklyle has quit IRC | 15:57 | |
| *** dklyle has joined #openstack-infra | 15:58 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add reenqueue utility https://review.opendev.org/613676 | 15:59 |
| *** hamzy has quit IRC | 16:01 | |
| *** gyee has joined #openstack-infra | 16:02 | |
| *** udesale has quit IRC | 16:06 | |
| *** ykarel|afk is now known as ykarel | 16:08 | |
| *** ociuhandu has joined #openstack-infra | 16:14 | |
| *** ociuhandu has quit IRC | 16:19 | |
| *** jamesmcarthur_ has joined #openstack-infra | 16:20 | |
| *** jamesmcarthur has quit IRC | 16:23 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 16:26 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 16:26 |
| *** whoami-rajat has quit IRC | 16:27 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 16:29 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 16:29 |
| *** ociuhandu has joined #openstack-infra | 16:30 | |
| clarkb | infra-root ^ I'm going to try and getting that stack green today if possible. Then hopefully we can switch over gitea01 to LE cert soon | 16:39 |
| clarkb | https://review.opendev.org/#/c/694182/ is the first bit of that stack and should be safe to land now though | 16:40 |
| kashyap | Folks, another random question - in the Gate, do we also support the "Eoan Ermine" or just the LTS (Bionic)? | 16:40 |
| kashyap | (Context for thers: the "Eoan Ermine" is the latest Ubuntu release) | 16:41 |
| ssbarnea | haha, my guess was right. | 16:41 |
| clarkb | kashyap: just LTS, intermediate releases are only supported for 9 months | 16:41 |
| fungi | kashyap: we only have lts ubuntu releases | 16:41 |
| fungi | so bionic is the latest (18.04 lts) | 16:42 |
| kashyap | fungi: Right, figured as much; thanks! | 16:42 |
| fungi | we'll have images for the "f" release next year | 16:42 |
| * kashyap goes to double-confirm if libvirt 4.0.0 and QEMU 2.11 are still the latest on Bionic (LTS) | 16:42 | |
| kashyap | [Context: I'm doing the "pick the next min libvirt/QEMU versions for V release" dance] | 16:43 |
| kashyap | [Email to the list upcoming...] | 16:43 |
| *** hamzy has joined #openstack-infra | 16:51 | |
| *** pkopec has quit IRC | 16:52 | |
| *** jamesmcarthur has joined #openstack-infra | 16:53 | |
| clarkb | kashyap: fwiw that decision should likely be made with openstack's supported distros and releases information more so than the available test platforms | 16:55 |
| kashyap | clarkb: Oh, certainly; I've done that a few times :-) | 16:55 |
| clarkb | that said the test platforms also reflect that | 16:55 |
| kashyap | clarkb: That's precisely what I'm doing :-) | 16:55 |
| kashyap | clarkb: I spent much of my today on duking around this topic; updated this: https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix | 16:56 |
| *** pgaxatte has quit IRC | 16:56 | |
| *** jamesmcarthur_ has quit IRC | 16:56 | |
| kashyap | clarkb: I'm just wondering if Ubuntu 'Bionic' gets access to newer bits via "Cloud Archive"? | 16:56 |
| clarkb | kashyap: it does, but I don't know if that necessarily includes libvirt updates too | 16:57 |
| *** ralonsoh has quit IRC | 16:58 | |
| *** Benibr has left #openstack-infra | 16:58 | |
| *** aedc has joined #openstack-infra | 16:59 | |
| *** jamesmcarthur has quit IRC | 17:01 | |
| *** jamesmcarthur has joined #openstack-infra | 17:01 | |
| *** ralonsoh has joined #openstack-infra | 17:01 | |
| kashyap | Hmm | 17:01 |
| *** jamesmcarthur has quit IRC | 17:01 | |
| kashyap | clarkb: The thing is - before I bump the agreed-upon libvirt/QEMU version for Ussuri, we need to pick the _next_ minimum versions | 17:02 |
| kashyap | ... and advertize them for operators. (Like we normally do) | 17:02 |
| *** lucasagomes has quit IRC | 17:03 | |
| jrosser | this is a bionic/train deployment which uses UCA https://pastebin.com/raw/NrBGKQDL | 17:03 |
| *** Lucas_Gray has quit IRC | 17:09 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 17:10 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 17:10 |
| kashyap | jrosser: Ah, thanks! | 17:13 |
| kashyap | jrosser: So Bionic _does_ have QEMU 4.0 -- do you have to enable anything special? | 17:13 |
| kashyap | jrosser: Because, this shows only QEMU 2.11 as the "latest": https://packages.ubuntu.com/bionic/qemu | 17:13 |
| jrosser | that won't be showing you wants in UCA sadly | 17:13 |
| jrosser | it's a totally seperate repo | 17:14 |
| kashyap | Ah-ha! Right | 17:14 |
| *** kjackal has quit IRC | 17:14 | |
| jrosser | in openstack-ansible we do this https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/vars/ubuntu-18.04.yml#L99 | 17:14 |
| kashyap | jrosser: Trying to find a better URL than this - http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/train/main/binary-i386/Packages | 17:15 |
| jrosser | and that will be the repo we use on master too for the whole U development cycle until the U version of UCA is released, which doesnt happen until after openstack itself releases U | 17:15 |
| *** iurygregory has quit IRC | 17:15 | |
| jrosser | so there is always a bit of chicken/egg there | 17:15 |
| kashyap | jrosser: Can you also post what's the libguestfs package version in Bionic/CloudArchive, please? | 17:16 |
| *** ociuhandu has quit IRC | 17:17 | |
| jrosser | I don't think i see that in UCA, on my compute nodes I have 1:1.36.13-1ubuntu3.3 coming from the main bionic repo | 17:18 |
| *** rpittau is now known as rpittau|afk | 17:18 | |
| jrosser | kashyap: you can browse it here http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/ | 17:19 |
| kashyap | jrosser: Yeah, just found it; but the "main" doesn't correlate to Bionic, does it? | 17:19 |
| frickler | kashyap: there are special pages that show the versions in UCA, including qemu and libvirt http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/train_versions.html | 17:19 |
| kashyap | frickler: Sweet; thanks! | 17:19 |
| kashyap | jrosser: I was expecting libguestfs 1.40; but 1.36 should also be fine | 17:20 |
| * jrosser bookmarks that | 17:20 | |
| kashyap | Thanks, both. Now /me goes to finish up the RFC email to the list | 17:20 |
| *** hamzy has quit IRC | 17:21 | |
| frickler | kashyap: if you also want to know plans for 20.04, best ask jamespage and/or coreycb, currently they are both here, but otherwise also in #ubuntu-server | 17:21 |
| *** hamzy has joined #openstack-infra | 17:22 | |
| kashyap | What does 20.04 map on to? | 17:24 |
| *** gfidente is now known as gfidente|afk | 17:25 | |
| *** ociuhandu has joined #openstack-infra | 17:28 | |
| fungi | that will be the "f" release | 17:31 |
| fungi | in april next year | 17:31 |
| *** jamesmcarthur has joined #openstack-infra | 17:32 | |
| AJaeger | config-core, could you review https://review.opendev.org/681582 and https://review.opendev.org/694570 , please? | 17:33 |
| AJaeger | clarkb: want to self-approve https://review.opendev.org/690718 - or change based on the last comment? | 17:34 |
| AJaeger | config-core, please also review https://review.opendev.org/#/c/693455/ | 17:35 |
| kashyap | fungi: Thanks | 17:39 |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: [WIP] Gitea: add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694776 | 17:39 |
| kashyap | AJaeger: A quick question; do you know a reliable URL to find the latest (shipped) QEMU / libvirt packages for openSUSE (and SLES)? | 17:40 |
| kashyap | For openSUSE, I found this - https://software.opensuse.org/package/qemu | 17:40 |
| frickler | 20.04 will be named "Focal Fossa" | 17:40 |
| *** ociuhandu has quit IRC | 17:40 | |
| AJaeger | kashyap: that is indeed good for openSUSE - and SLES 15 and openSUSE 15.1 should have the same version in general. | 17:41 |
| AJaeger | kashyap: for SLES, you need a subscription and then can search on scc.suse.com | 17:41 |
| kashyap | AJaeger: Hmm, most latest distros support at least QEMU 4.0 (or above); and libvirt 5.0 (or above) | 17:42 |
| AJaeger | kashyap: let me check quickly for you | 17:42 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Gitea: add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694817 | 17:42 |
| AJaeger | kashyap: SLES 15 SP1: libvirt 5.1.0; qemu 3.1.1 | 17:43 |
| kashyap | AJaeger: Right, that info is already here - https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix#Distro_minimum_versions | 17:44 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Change issue tracking for airshipctl https://review.opendev.org/694791 | 17:44 |
| AJaeger | so, what else do you need? ;) | 17:44 |
| kashyap | I was trying to settle on "next_min" versions for libvirt/QEMU as 5.0.0 (libvirt) and 4.0.0 (QEMU) | 17:44 |
| AJaeger | what timeframe? | 17:44 |
| kashyap | AJaeger: For "V" release | 17:45 |
| AJaeger | that would mean SLES 15 SP2 which is not released yet... | 17:45 |
| *** jpena is now known as jpena|off | 17:47 | |
| AJaeger | kashyap: let's assume that for V release SLES will be fine with qemu 4 | 17:48 |
| *** kjackal has joined #openstack-infra | 17:48 | |
| kashyap | Yeah, that's a very reasoable guess | 17:48 |
| kashyap | AJaeger: I'm writing email to the list; almost done -- please respond there? | 17:48 |
| AJaeger | kashyap: I can - if none of my colleagues beats me to it ;) | 17:49 |
| AJaeger | thanks, kashyap | 17:49 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Gitea: add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694817 | 17:57 |
| roman_g | AJaeger: thank you for the review. I know it's (external tracker url) not supported. Trying to add it, work in progress. | 18:00 |
| *** dtantsur is now known as dtantsur|afk | 18:03 | |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Change issue tracking for airshipctl https://review.opendev.org/694791 | 18:03 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 18:05 | |
| kashyap | AJaeger: There we go - http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010907.html | 18:12 |
| kashyap | ("On next minimum libvirt / QEMU versions for "V" release") | 18:12 |
| * kashyap --> cooking | 18:12 | |
| *** ykarel is now known as ykarel|away | 18:15 | |
| *** ykarel|away has quit IRC | 18:20 | |
| *** igordc has joined #openstack-infra | 18:46 | |
| *** gouthamr_ is now known as gouthamr | 18:51 | |
| *** tesseract has quit IRC | 18:52 | |
| *** ralonsoh has quit IRC | 18:56 | |
| *** diablo_rojo has joined #openstack-infra | 18:56 | |
| *** weshay has joined #openstack-infra | 19:00 | |
| *** diablo_rojo has quit IRC | 19:04 | |
| *** diablo_rojo has joined #openstack-infra | 19:04 | |
| openstackgerrit | Roman Gorshunov proposed opendev/system-config master: Gitea: add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694776 | 19:05 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Gitea: add possibility to add arbitrary URL pointing to issues https://review.opendev.org/694817 | 19:06 |
| *** diablo_rojo has quit IRC | 19:11 | |
| *** diablo_rojo has joined #openstack-infra | 19:12 | |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Add v2 branch for monitoring by IRC gerritbot https://review.opendev.org/694834 | 19:13 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Add v2 branch for monitoring by IRC gerritbot https://review.opendev.org/694834 | 19:14 |
| clarkb | I've apparently neglected to account for DST change when scheduling a 10am local dentist visit tomorrow. I thought the meeting would be Noon time but is actually now at 11am post DST disablement | 19:18 |
| clarkb | infra-root ^ there is a good chance I'll be done with the dentisting by 11 but likely not home by then to run the meeting | 19:18 |
| *** dpawlik has joined #openstack-infra | 19:26 | |
| *** diablo_rojo has quit IRC | 19:26 | |
| clarkb | pabelanger: dmsimard: what is a good way to do what https://review.opendev.org/#/c/694181/10/playbooks/roles/letsencrypt-create-certs/handlers/restart_gitea.yaml lines 28-34 try to do without needing to install extra libs? | 19:26 |
| clarkb | basically I'm trying to check if the service is already running (either via ps or docker or similar) so that the service can be restarted if it is running but ignored otherwise | 19:27 |
| clarkb | pids needs psutil and docker stuff needs some docker python libn | 19:27 |
| clarkb | maybe I should just command a ps | 19:27 |
| *** diablo_rojo has joined #openstack-infra | 19:29 | |
| *** aedc has quit IRC | 19:29 | |
| dmsimard | clarkb: what's the use case if gitea is not running ? shouldn't it always be running ? | 19:29 |
| dmsimard | I mean, if it shouldn't be running, perhaps that should be a variable toggle (i.e, gitea_enable_service: true/false) and then you use that variable to tell whether or not it needs to be started/restarted | 19:30 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 19:31 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 19:31 |
| fungi | clarkb: or grep /proc/[0-9]*/cmdline | 19:31 |
| clarkb | dmsimard: if it isn't running then either the service hasn't been installed yet (initial bootstrap of the system) or it has manually been stopped and I don't want to override admin actions | 19:31 |
| clarkb | dmsimard: and no I don't think we want a toggle | 19:31 |
| clarkb | beacuse they doesn't handle initial bootstrapping and makes "I need to disable this easily" way more difficult | 19:32 |
| clarkb | fungi: ya I went with pgrep which is essentially that | 19:32 |
| fungi | mordred: rackspace fanatical support just notified us of a host outage impacting an instance named "mttest" in dfw, so guessing that's something of yours | 19:33 |
| *** tosky has quit IRC | 19:34 | |
| clarkb | fwiw the apache stuff uses a similar pattern but since apache is run as a normal service we use service facts to check if it is enabled and running | 19:39 |
| clarkb | I think this mostly approximates that for docker + gitea | 19:40 |
| *** michael-beaver has joined #openstack-infra | 19:41 | |
| *** kjackal_v2 has joined #openstack-infra | 19:45 | |
| *** kjackal has quit IRC | 19:46 | |
| mordred | fungi: yes - that is something I spun up to test some of the docker/dib stuff - it will not kill anything if it needs to just be deleted or whatnot | 19:51 |
| fungi | they'll probably just cold-migrate it | 19:53 |
| mordred | that is acceptable | 19:55 |
| *** Vadmacs has quit IRC | 20:01 | |
| *** eharney has quit IRC | 20:02 | |
| ianw | mordred: so the latest effort can build an image in nodepool-builder container | 20:05 |
| ianw | https://zuul.opendev.org/t/zuul/build/b7dd21b7ebdb4d279bcb22ba6125cdfd/log/nodepool/builds/test-image-0000000001.log <- first gate containerised dib build that resulted in a .qcow2 | 20:06 |
| ianw | it hasn't uploaded, instantiated or booted, but it's a start :) | 20:06 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org cert with LE https://review.opendev.org/694181 | 20:07 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 20:07 |
| ianw | mordred: not sure if you saw my note, but to use the speculative images i had to -> "image: zuul-jobs.buildset-registry:5000/zuul/nodepool-builder" | 20:07 |
| clarkb | I think ^ is really close now would not be surprised if that passed testing this time | 20:07 |
| ianw | i didn't see that documented anywhere? | 20:07 |
| ianw | clarkb: cool! will look ... is there a tl;dr on the issues? | 20:07 |
| ianw | creating certs before starting services i imagine? | 20:08 |
| clarkb | ianw: it was the stuff I worried about last week. Basically need to handle the "/var/gitea/certs does not exist yet on bootstrap and docker isn't there to restart the service" | 20:08 |
| clarkb | ya | 20:08 |
| clarkb | I've now got it creating the dir if necessary and checking if the service is running before trying to restart it | 20:08 |
| AJaeger | config-core, could you review https://review.opendev.org/#/c/693455/ , https://review.opendev.org/681582 and https://review.opendev.org/694570 , please? | 20:09 |
| ianw | clarkb: cool; another thought is that if docker isn't there to restart, then gitea wouldn't be active to restart either? as long as the pgrep keeps working, LGTM though :) | 20:10 |
| clarkb | AJaeger: I WIPed https://review.opendev.org/#/c/690718/1 | 20:10 |
| clarkb | AJaeger: with a note if you want details | 20:10 |
| clarkb | AJaeger: can you check my comment on https://review.opendev.org/#/c/681582/4 | 20:12 |
| clarkb | mordred: maybe you can review https://review.opendev.org/#/c/693455/ as it adds a podman install role and you've probably got the most exposure to that? | 20:14 |
| *** diablo_rojo has quit IRC | 20:19 | |
| *** dpawlik has quit IRC | 20:22 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove lib-forward-testing-plugin https://review.opendev.org/694570 | 20:24 |
| *** dpawlik has joined #openstack-infra | 20:27 | |
| *** roman_g has quit IRC | 20:31 | |
| *** roman_g has joined #openstack-infra | 20:32 | |
| mordred | clarkb: cool, looking: https://review.opendev.org/#/c/630406/69/playbooks/roles/install-podman/README.rst is a podman role I have in a different patch | 20:38 |
| mordred | clarkb: that one from ianw looks good for zuul-jobs - installing the ppa using apt_repository there seems like the right call | 20:40 |
| *** dpawlik has quit IRC | 20:41 | |
| *** roman_g has quit IRC | 20:41 | |
| *** roman_g has joined #openstack-infra | 20:43 | |
| *** dpawlik has joined #openstack-infra | 20:44 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Manage opendev.org with LE on all giteas https://review.opendev.org/694184 | 20:47 |
| clarkb | I think ^ that stack should be ready for review now infra-root | 20:47 |
| clarkb | the gitea test passed on the first change that updates only gitea01 | 20:47 |
| clarkb | hrm the mirror job timed out https://zuul.opendev.org/t/openstack/build/1fb2347e85684dafb4e3fca808064c47/log/job-output.txt any idea if that is "normal"? | 20:51 |
| openstackgerrit | Merged zuul/zuul-jobs master: Role to install podman https://review.opendev.org/693455 | 20:52 |
| mordred | clarkb: what's the story with gitea99? | 20:52 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Add "external-tracker-url" key to projects.yaml linter script https://review.opendev.org/694817 | 20:53 |
| *** kjackal_v2 has quit IRC | 20:54 | |
| clarkb | mordred: https://review.opendev.org/#/c/672354/4/playbooks/zuul/templates/group_vars/gitea-lb.yaml.j2 it was added in that change because we couldn't disable gitea01 in production without breaking testing | 20:54 |
| clarkb | mordred: because disabling it in production applied the same disablement to gitea01 in testing and then haproxy would fail due to no backend service running | 20:54 |
| mordred | clarkb: ah - gotcha | 20:57 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add gitea acme records https://review.opendev.org/694182 | 20:58 |
| mordred | clarkb: stack looks good - assuming that one worked | 20:58 |
| clarkb | cool I think once we get gitea01 rolled out and happy we should be safe to deploy the second change too. | 21:00 |
| mordred | ++ | 21:01 |
| *** goldyfruit_ has quit IRC | 21:02 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile : install sudo for nodepool-builder https://review.opendev.org/694709 | 21:03 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 21:03 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add an ENABLE_DEBUG build arg/environment https://review.opendev.org/694845 | 21:03 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: [WIP] Change issue tracking system for airshipctl https://review.opendev.org/694791 | 21:03 |
| * paladox apparently on the Gerrit Frontend Team now! | 21:05 | |
| *** eharney has joined #openstack-infra | 21:12 | |
| mordred | pabelanger: \o/ | 21:14 |
| mordred | gah | 21:14 |
| mordred | paladox: \o/ | 21:14 |
| mordred | pabelanger: I mean, I'll happily give you a \o/ any time - but in this case it was aimed at paladox and I just missed | 21:15 |
| paladox | :P | 21:15 |
| paladox | I got sent a picture of the slide from the hackathon listing me under the team! | 21:15 |
| *** goldyfruit has joined #openstack-infra | 21:17 | |
| clarkb | I'm going to find food while the gitea changes are retested. If that mirror job times out again I guess I'll actually look into it next | 21:18 |
| *** kjackal has joined #openstack-infra | 21:24 | |
| *** dpawlik has quit IRC | 21:29 | |
| *** rkukura has joined #openstack-infra | 21:32 | |
| *** kjackal has quit IRC | 21:36 | |
| *** kjackal has joined #openstack-infra | 21:36 | |
| *** ociuhandu has joined #openstack-infra | 21:38 | |
| *** dpawlik has joined #openstack-infra | 21:43 | |
| *** ociuhandu has quit IRC | 21:43 | |
| *** michael-beaver has quit IRC | 21:56 | |
| *** rfolco has quit IRC | 22:04 | |
| ianw | 2019-11-18 20:55:56.897487 | bridge.openstack.org | changed: All items complete | 22:06 |
| ianw | 2019-11-18 20:55:56.897637 | | 22:06 |
| ianw | 2019-11-18 21:19:14.359157 | RUN END RE | 22:06 |
| ianw | clarkb: ^ yeah, the playbook just ... stopped? | 22:06 |
| *** dpawlik has quit IRC | 22:08 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add an ENABLE_DEBUG build arg/environment https://review.opendev.org/694845 | 22:20 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 22:20 |
| *** darvon has quit IRC | 22:20 | |
| *** darvon has joined #openstack-infra | 22:20 | |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 22:21 |
| *** darvon has quit IRC | 22:22 | |
| *** darvon has joined #openstack-infra | 22:23 | |
| *** hwoarang has quit IRC | 22:24 | |
| *** kjackal has quit IRC | 22:25 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP test if run-mirror job always times out https://review.opendev.org/694851 | 22:25 |
| clarkb | ianw: ^ that might help provide more clues /me looks at logs while waiting for that to finish | 22:25 |
| *** tosky has joined #openstack-infra | 22:26 | |
| *** pcaruana has quit IRC | 22:26 | |
| *** kaisers1 has joined #openstack-infra | 22:28 | |
| tonyb | ianw, clarkb: I think I've hit that git issue again | 22:28 |
| *** hwoarang has joined #openstack-infra | 22:28 | |
| ianw | tonyb: hrm ... afaik gitea06 is still out of rotation, there wasn't much of an update on the gitea issue | 22:28 |
| ianw | tonyb: same IP? | 22:29 |
| tonyb | ianw: tony 5608 0.0 0.8 475788 143596 pts/6 S+ Nov18 0:01 | | \_ /usr/libexec/git-core/git fetch-pack --stateless-rpc --stdin --lock-pack --include-tag --thin https://opendev.org/openstack/nova/ | 22:29 |
| *** kaisers has quit IRC | 22:29 | |
| tonyb | My clone has been 'running' for >24hours | 22:29 |
| tonyb | ianw: let me check | 22:29 |
| tonyb | ianw: 121.45.215.66 ... I don't knwo if that's the same | 22:30 |
| pabelanger | clarkb: systemd, state: restarted should be what you want | 22:30 |
| pabelanger | IIRC, that only restarts, if running | 22:30 |
| tonyb | ianw: if you have some time in 20ish mins I can do some debug with you | 22:31 |
| *** mriedem has quit IRC | 22:31 | |
| ianw | Nov 18 01:36:21 gitea-lb01 haproxy[6]: ::ffff:121.45.215.66:36366 [18/Nov/2019:01:36:20.691] balance_git_https balance_git_https/gitea06.opendev.org 1/0/410 7176 -- 42/42/41/8/0 0/0 | 22:31 |
| ianw | tonyb: sure, will be here :) | 22:32 |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 22:32 |
| tonyb | ianw: Oh so it's back in rotation | 22:32 |
| ianw | maybe some automation turns it back on, unless an infra-root did? | 22:32 |
| corvus | paladox: yeah, they seemed pretty happy to have a non-googler helping out there :) also, are you aware of the polygerrit frontend test plugin for chromium? it looked super useful. | 22:32 |
| clarkb | pabelanger: it isn't a systemd service | 22:33 |
| paladox | corvus yup, i helped tao test it! | 22:33 |
| clarkb | ianw: possible a docker image upadte for haproxy happened and so state was reread off disk | 22:33 |
| clarkb | ianw: https://zuul.opendev.org/t/openstack/build/5857d770701243dbb5e501741fe1fdc9/log/bridge.openstack.org/syslog.txt.gz#1513 the mirror job seems to do a reboot? | 22:33 |
| pabelanger | clarkb: service should work too | 22:33 |
| clarkb | pabelanger: it isn't a service | 22:34 |
| pabelanger | oh | 22:34 |
| clarkb | pabelanger: that is the issue. The case where it is a system service is already handled that way | 22:34 |
| clarkb | in any case the pgrep seems to be working. Now to figure out why this other job is unhappy | 22:34 |
| ianw | clarkb: huh, i don't think so?! | 22:35 |
| clarkb | ianw: curiously that happens on the bridge node too | 22:35 |
| paladox | corvus it's a awesome tool that makes testing easier (since you can edit the html/js files live) | 22:35 |
| clarkb | ianw: tonyb we should manually remove it again then push a change up to pull 06 from the backends in the ansible var data | 22:38 |
| ianw | clarkb: yeah ... i was hoping for more comment in the gitea issue | 22:39 |
| ianw | https://github.com/go-gitea/gitea/issues/9006 | 22:39 |
| ianw | it has become clear that the error is "fatal: the remote end hung up unexpectedly" | 22:39 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: blacklist pipeline names that can not be used in template https://review.opendev.org/693961 | 22:40 |
| ianw | which makes me wonder if the LB is having trouble talking to gitea06? | 22:40 |
| clarkb | ianw: that could be, we should see errors in the haproxy log for gitea06 if that happens I think | 22:40 |
| ianw | Nov 18 01:36:21 gitea-lb01 haproxy[6]: ::ffff:121.45.215.66:36366 [18/Nov/2019:01:36:20.691] balance_git_https balance_git_https/gitea06.opendev.org 1/0/410 7176 -- 42/42/41/8/0 0/0 | 22:40 |
| clarkb | of course its liek a single character code | 22:40 |
| ianw | yeah, from my parsing of that, it is not showing an error | 22:40 |
| ianw | howerver, it is ipv6 ... | 22:40 |
| clarkb | isn't that actually ipv4? | 22:41 |
| ianw | yeah it must be, there's not aaaa for gitea hosts | 22:41 |
| ianw | dunno why it shows it like that | 22:41 |
| ianw | there's no ping packet loss gitea-lb01 -> gitea06 | 22:41 |
| clarkb | that syslog stuff is a user session shutdown not host shutdown | 22:42 |
| clarkb | is it possible that tonyb's connection isn't pulling bits fast enough and is hitting a timeout on the haproxy side and for some reason git doesn't notice it? | 22:43 |
| tonyb | clarkb: are you saying that my connect is slow?! | 22:44 |
| ianw | clarkb: the thing is, gitea06 is full of the message | 22:44 |
| ianw | it does seem that tonyb's ip hashes to gitea06 maybe ... he constantly goes there | 22:45 |
| *** rcernin has joined #openstack-infra | 22:46 | |
| tonyb | Happy to play ... and or tarball up my repo for y'all to play | 22:47 |
| clarkb | yes we do ip pinning otherwise git clone breaks for other reasons | 22:47 |
| clarkb | tonyb: ianw does cloning directly to 06 work? | 22:47 |
| ianw | clarkb: from memory, it did not replicate when tony went directly | 22:48 |
| ianw | but i'm not sure we could replicate it | 22:48 |
| tonyb | clarkb: It did last week | 22:48 |
| tonyb | clarkb: so it seems to be somethign about upload-pack and the delta between my last update and today | 22:49 |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 22:49 |
| clarkb | tonyb: oh you are not cloning but doing a fetch update? | 22:49 |
| clarkb | fwiw that would be similar to the problem we do ip based balancing for | 22:50 |
| *** slaweq has quit IRC | 22:50 | |
| clarkb | I wonder if the local state was populated from some upstream state (possibly not from 06) which now conflicts with 06's state somehow | 22:50 |
| clarkb | resulting in unresolvable git packaing or something | 22:51 |
| clarkb | ianw: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/tasks/main.yaml#L80 is the last thing to run in that run-mirror job ansible | 22:51 |
| clarkb | ianw: specifically that symlink. syslog doesn't show it getting to the others | 22:51 |
| ianw | clarkb: here's a strace of the dying process i extracted: https://github.com/go-gitea/gitea/files/3861071/16131.trace.txt | 22:51 |
| ianw | clarkb: i.e. openafs not working? | 22:52 |
| ianw | that would ... suck | 22:52 |
| ianw | i wonder if upstream kernel has updated in the last little bit | 22:53 |
| clarkb | ianw: https://zuul.opendev.org/t/openstack/build/5857d770701243dbb5e501741fe1fdc9/log/mirror01.openafs.provider.opendev.org/syslog.txt.gz#1430 ya thats the relevant syslog bit | 22:54 |
| clarkb | tonyb: your client doesn't crash though? | 22:55 |
| clarkb | reading that strace it does seem like it points to the proxy if the client is still happily sitting there | 22:55 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: config: blacklist pipeline names that can not be used in template https://review.opendev.org/693961 | 22:55 |
| ianw | 4.15.0-70-generic (buildd@lgw01-amd64-055) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 | 22:55 |
| openstackgerrit | Merged zuul/zuul-registry master: Disable namespacing https://review.opendev.org/689219 | 22:57 |
| clarkb | tonyb: also does it work to do a clean clone and only fail on that delta update? | 22:58 |
| tonyb | clarkb: Yes, but I wasn't able to reset the repo to some point in the past and get the update to fail | 23:02 |
| ianw | clarkb: openafs 1.8.4 looks like it has quite a few fixes. 1.8.5 not so much ... we could try updating? | 23:03 |
| clarkb | tonyb: and your local repo fscks cleaning? | 23:03 |
| clarkb | er cleanly | 23:03 |
| clarkb | ianw: are we able to update and test it in the jobs prior to having that deploy on our mirrors? | 23:04 |
| clarkb | if so that seems like a reasonable next step | 23:04 |
| ianw | clarkb: that's a good question. i don't think we pin the versions in the ansible | 23:04 |
| openstackgerrit | Merged zuul/zuul master: Fix issue search in FakeGithubClient https://review.opendev.org/694553 | 23:04 |
| clarkb | tonyb: something like `git fsck --full --strict` | 23:05 |
| *** tkajinam has joined #openstack-infra | 23:09 | |
| clarkb | run-mirror timeouts happen with no changes to our code, https://review.opendev.org/#/c/694851/ that rules out my minor changes to the le stuff causing problems in the opendev le change | 23:11 |
| ianw | openafs hanging makes sense | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| clarkb | tonyb: ianw another thing it could be is behavior differences between gitea prior to our upgrade at the summit which used git proper and now they use go-git | 23:11 |
| ianw | clarkb: but why only gitea06? | 23:12 |
| clarkb | some corner case in pack handling across that transition maybe | 23:12 |
| clarkb | ianw: because that is the only one tonyb talks to with the triggering state local repo? | 23:12 |
| ianw | i also deleted all the repos on disk, and recreated it | 23:12 |
| clarkb | if tonyb's IP changed it is possible we would see the same thing against another backend | 23:12 |
| clarkb | or did we confirm it works after removing 06 from the rotation? | 23:12 |
| ianw | clarkb: this is happening multiple times/second on gitea06 ... i think possibly he just notices | 23:12 |
| clarkb | ah | 23:13 |
| clarkb | 06 has significantly more connection requests coming in than 01 according to cacti | 23:14 |
| clarkb | 06 has the highest out of all 8 | 23:15 |
| clarkb | I wonder if that is where the RH nat maps too | 23:16 |
| clarkb | but maybe it is a load problem and we are finding the flaw with the IP hashed balance method? | 23:16 |
| ianw | i didn't see any oom or similar, however | 23:16 |
| clarkb | ya and resource consumption looks ok on 06 | 23:16 |
| *** slaweq has quit IRC | 23:17 | |
| openstackgerrit | Merged zuul/zuul master: Add propagate=0 for loggers in logging.conf-sample https://review.opendev.org/694170 | 23:18 |
| *** jaosorior has quit IRC | 23:21 | |
| clarkb | if tonyb can confirm a clean fsck on the local repo then ya my next best guess is somethign to do with the host and/or networking between it and the load balancer. Given that removing it from services and rebuilding it is probably worth a shot | 23:22 |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: DNM: Test zuul-stream-functional with ara 1.2 https://review.opendev.org/694622 | 23:23 |
| tonyb | clarkb: running an fsck now | 23:25 |
| tonyb | clarkb: I can archive my repo and then try against other backends directly | 23:25 |
| clarkb | tonyb: that might help too as we'd be able to see if it is backend specific better that way | 23:26 |
| tonyb | ianw, clarkb: fsck completed I have 2 dangling objects but I doubt that's it | 23:26 |
| *** dchen has joined #openstack-infra | 23:27 | |
| clarkb | tonyb: maybe create archive of repo in its current state then prune what you've got and try to update? that should tell us if the dangling objects are at fault (pruning should remove them) | 23:28 |
| tonyb | clarkb: sure | 23:29 |
| clarkb | oh man page says you should probably run git gc instead of prune | 23:29 |
| tonyb | clarkb: which I did because #oldskool ;P | 23:30 |
| tonyb | Hmm it didn't drop the dangling commits | 23:30 |
| *** ociuhandu has joined #openstack-infra | 23:31 | |
| tonyb | ok they're gone now | 23:32 |
| tonyb | still 'hangs' | 23:33 |
| *** kjackal has joined #openstack-infra | 23:33 | |
| tonyb | as does gitea01 | 23:33 |
| openstackgerrit | Merged zuul/zuul master: Revert "Revert "Add the process environment to zuul.conf parser"" https://review.opendev.org/692469 | 23:33 |
| *** ociuhandu has quit IRC | 23:36 | |
| ianw | gitea01 has plenty of these messages too .. | 23:36 |
| *** goldyfruit has quit IRC | 23:36 | |
| clarkb | ok if it isn't host specific then that may point back to possible go-git gitea change bug/interaction | 23:37 |
| ianw | clarkb: i'm trying to build openafs-1.8.5 packages now. as an emergency fix, we could put them in a different part of the PPA and update the jobs to use that, if it works, put them in the regular ppa | 23:37 |
| ianw | we can think about pinning things when we have less issues on the go :/ | 23:38 |
| clarkb | ianw: if that is simple it sounds good to me. You can update https://review.opendev.org/#/c/694851/ to test it too if you want | 23:38 |
| tonyb | Anything else to try? | 23:39 |
| *** diablo_rojo has joined #openstack-infra | 23:40 | |
| tonyb | I can trivially wait until y'all are done with the AFS emergency | 23:40 |
| clarkb | let me make sure I've got the current state correct. We have ruled out the proxy because you can reproduce via direct communication to the backend. We have ruled out host specific problems (likely) because multiple hosts exhibit this. We have ruled out local git repo problems as git gc'd repo also exhibits the problem. | 23:41 |
| openstackgerrit | Merged zuul/zuul master: executor: add merge_jobs options to disable gearman merge jobs https://review.opendev.org/676974 | 23:41 |
| clarkb | In that case my hunch is that we are dealing with a gitea issue and the bug ianw files may be the best chance at figuring it out? | 23:42 |
| tonyb | clarkb: Yes that sounds like a correct summary | 23:42 |
| tonyb | clarkb: I can *clone* fine either direct or via the LB | 23:43 |
| tonyb | I can easily just *not* update nova until we have more ideas | 23:43 |
| *** jamesmcarthur has quit IRC | 23:46 | |
| *** jamesmcarthur has joined #openstack-infra | 23:46 | |
| *** jamesmcarthur has quit IRC | 23:48 | |
| *** gfidente|afk has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!