| *** mlavalle has quit IRC | 00:04 | |
| *** slaweq has quit IRC | 00:26 | |
| *** jhesketh has joined #opendev | 00:52 | |
| *** kevinz has joined #opendev | 01:23 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 01:36 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 01:54 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 02:31 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 02:50 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 03:10 |
| *** yuri has joined #opendev | 03:27 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 03:27 |
| *** yuri has quit IRC | 03:31 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 03:57 |
| *** ykarel|away is now known as ykarel | 04:57 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 05:03 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 05:28 |
| *** ysandeep|away is now known as ysandeep | 05:46 | |
| *** dpawlik has joined #opendev | 06:02 | |
| *** redrobot has quit IRC | 06:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 06:23 |
| *** slaweq has joined #opendev | 06:47 | |
| *** rpittau|afk is now known as rpittau | 06:51 | |
| *** hashar has joined #opendev | 07:05 | |
| hrw | morning | 07:12 |
| openstackgerrit | Marcin Juszkiewicz proposed opendev/base-jobs master: add arm64 nodesets https://review.opendev.org/728810 | 07:18 |
| hrw | centos-8, debian-buster and ubuntu-focal for arm64 | 07:19 |
| hrw | so there will be no need to define them each time | 07:19 |
| openstackgerrit | Simon Westphahl proposed openstack/diskimage-builder master: Use kpartx option to update partition mappings https://review.opendev.org/728824 | 07:21 |
| hrw | sent job for ubuntu-focal-arm64. will see how it goes | 07:24 |
| *** mnasiadka has quit IRC | 07:25 | |
| *** mnasiadka has joined #opendev | 07:27 | |
| hrw | ok. NODE_FAILURE | 07:30 |
| ianw | hrm, did it build? | 07:45 |
| ianw | https://nb03.openstack.org/ubuntu-focal-arm64-0000000445.log | 07:45 |
| ianw | no .... and i know what's wrong ... that isn't deployed with our new container builder (buster era) yet | 07:46 |
| ianw | the debootstrap on it is too old | 07:46 |
| ianw | i didn't think of that, bummer | 07:46 |
| ianw | we know it does work on our recent nodes | 07:47 |
| ianw | before we take any drastic action, it's probably a question for mordred on where we're at with replacing the arm64 builder with our docker containers | 07:47 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] generate ssl check list directly from letsencrypt variables https://review.opendev.org/728743 | 07:53 |
| *** tosky has joined #opendev | 07:56 | |
| *** lpetrut has joined #opendev | 08:01 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** tkajinam has quit IRC | 08:02 | |
| hrw | ianw: ok | 08:02 |
| hrw | ianw: I built focal image using dib on buster | 08:03 |
| *** ysandeep is now known as ysandeep|lunch | 08:21 | |
| hrw | ianw: https://review.opendev.org/728798 (openstack/requirements) waits for ubuntu-focal-arm64 node | 08:24 |
| hrw | ianw: it will be using it to make sure that each python package present there is available on aarch64. | 08:25 |
| *** panda|off is now known as panda | 08:34 | |
| hrw | ianw: debootstrap from buster-backports knows focal | 08:54 |
| hrw | ianw: that's why it worked for me | 08:54 |
| *** DSpider has joined #opendev | 08:57 | |
| *** ykarel is now known as ykarel|lunch | 09:07 | |
| *** yuri has joined #opendev | 09:16 | |
| *** hashar is now known as hasharAway | 09:25 | |
| ianw | hrw: yeah, unfortunately the arm builder nb03.openstack.org is xenial | 09:26 |
| ianw | we have work in progress to deploy a new containerised version, that will work with focal | 09:27 |
| hrw | omg. | 09:27 |
| hrw | xenial. | 09:27 |
| *** yuri has quit IRC | 09:30 | |
| hrw | ianw: any patch in review I could track? | 09:30 |
| hrw | debootstrap (1.0.78+nmu1ubuntu1.10) xenial; urgency=medium | 09:30 |
| hrw | * Add (Ubuntu) focal as a symlink to gutsy. (LP: #1848716) | 09:31 |
| openstack | Launchpad bug 1848716 in debootstrap (Ubuntu) "Add Ubuntu Focal as a known release" [High,Fix released] https://launchpad.net/bugs/1848716 - Assigned to Łukasz Zemczak (sil2100) | 09:31 |
| hrw | ianw: update packages? :D | 09:31 |
| ianw | it has debootstrap 1.0.114~bpo16.04+1 | 09:32 |
| hrw | ~bpo suggests Debian by hand build | 09:32 |
| ianw | yeah ... we have a backport, i can't quite remember why, i'm sure there's a comment | 09:33 |
| hrw | 1.0.114 is buster version | 09:33 |
| ianw | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-04-03.log.html#t2019-04-03T21:25:50 | 09:34 |
| ianw | so we backported the buster version to build buster | 09:36 |
| hrw | I see | 09:37 |
| ianw | i can probably do the same -- link gutsy to focal | 09:39 |
| *** sshnaidm|afk is now known as sshnaidm | 09:40 | |
| hrw | should be enough | 09:40 |
| hrw | focal was added in 117 and does not look like much changed between 114 and 117 | 09:41 |
| hrw | ianw: what time is at your place? Trying to keep a list of timezone of people I work with | 09:42 |
| ianw | i'm in melbourne .au ... 19:42 | 09:42 |
| hrw | thanks | 09:43 |
| hrw | I was not sure which of australian timezones to add | 09:44 |
| *** ysandeep|lunch is now known as ysandeep | 09:51 | |
| *** ykarel|lunch is now known as ykarel | 09:53 | |
| ianw | hrw: ok ... it's building ... https://launchpad.net/~openstack-ci-core/+archive/ubuntu/debootstrap/+sourcepub/11302190/+listing-archive-extra | 09:54 |
| hrw | yay | 09:54 |
| hrw | thank Ian | 09:54 |
| ianw | don't thank me till it works :) | 09:56 |
| hrw | ;) | 09:56 |
| hrw | ianw: https://review.opendev.org/#/c/728810/ would be nice addon for it ;d | 09:59 |
| ianw | hrw: i don't know why i didn't just symlink it on the host ... which i've done now | 10:06 |
| ianw | https://nb03.openstack.org/ubuntu-focal-arm64-0000000566.log is the build ... it's got further | 10:06 |
| ianw | debootstrap - 1.0.114~bpo16.04+2 is published, so everything's in sync | 10:07 |
| ianw | bummer, it looks like it hits some other package error | 10:08 |
| ianw | i'm afraid i'm out of time to debug | 10:08 |
| ianw | the real solution here is to get the arm64 builder upgraded | 10:08 |
| ianw | as soon as mordred is online, i'm sure he can update you on specifics | 10:08 |
| openstackgerrit | Jens Harbott (frickler) proposed opendev/system-config master: Document the need to use sudo in order to access OSC https://review.opendev.org/729196 | 10:10 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: DNM: test https://review.opendev.org/728640 | 10:11 |
| hrw | ianw: thanks for all the work. | 10:17 |
| hrw | ianw: have a nice evening | 10:17 |
| *** tkajinam has joined #opendev | 10:30 | |
| *** rpittau is now known as rpittau|bbl | 10:31 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 10:57 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 11:05 |
| *** roman_g has joined #opendev | 11:25 | |
| *** hasharAway is now known as hashar | 11:51 | |
| *** sean-k-mooney has quit IRC | 11:53 | |
| *** rpittau|bbl is now known as rpittau | 12:04 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Don't require tox_envlist https://review.opendev.org/726829 | 12:32 |
| *** ysandeep is now known as ysandeep|brb | 12:44 | |
| openstackgerrit | Merged opendev/irc-meetings master: Update OSH meeting chair and agenda link https://review.opendev.org/729017 | 13:01 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| *** ysandeep|brb is now known as ysandeep | 13:22 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 13:37 |
| *** lpetrut has quit IRC | 13:52 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 13:57 |
| *** mlavalle has joined #opendev | 13:57 | |
| *** hashar has quit IRC | 13:58 | |
| openstackgerrit | Oleksandr Kozachenko proposed zuul/zuul-jobs master: Add DaemonSet check for wait-for-pods role https://review.opendev.org/728503 | 14:00 |
| *** tkajinam has quit IRC | 14:11 | |
| *** tkajinam has joined #opendev | 14:11 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 14:26 |
| openstackgerrit | Bernard Cafarelli proposed openstack/project-config master: Update neutron stable grafana dashboards https://review.opendev.org/729291 | 14:42 |
| *** ykarel|afk is now known as ykarel | 14:43 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 15:01 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 15:12 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 15:15 |
| hrw | mordred: ianw mentioned that you may know what is a status of arm64 image builder upgrade | 15:19 |
| hrw | mordred: current one is xenial based and even with debootstrap update is unable to build focal image. My arm64 box runs buster and builds focal image with d-i-b without problem | 15:20 |
| mordred | yah | 15:20 |
| mordred | hrw: it's ... we're getting closer. the other builders are all running in docker, so we needed to get multi-arch docker image support working. that's 90% working, but the other day we found an issue where the multi-arch docker image builder backend seemed to be pulling inappropriate layers and thus making bad images | 15:21 |
| hrw | mordred: ~5h ago in backlog | 15:21 |
| mordred | hrw: so, once we've figured out what's wrong there, we should be good | 15:21 |
| hrw | mordred: yeah. | 15:21 |
| mordred | (and we'll incidentally have arm64 docker images of nodepool, which will be neat) | 15:22 |
| mordred | I need to do some more debugging of the underlying issue though - make sure we understand it | 15:22 |
| mordred | corvus: ^^ I *just* had a hunch about what might be happening there | 15:23 |
| clarkb | mordred: and the toolchain we are using is docker buildx as well as skopeo? | 15:23 |
| hrw | mordred: thanks | 15:23 |
| mordred | clarkb: yeah - but I think we observed the issue purely in buildx | 15:23 |
| corvus | mordred: o/ | 15:24 |
| mordred | corvus: my hunch is unfounded | 15:24 |
| corvus | boo | 15:24 |
| mordred | corvus: yeah. I was thinking it had to do with our workaround for push races | 15:25 |
| mordred | but we do all arches in step 1 | 15:25 |
| mordred | (I was thinking maybe if we were doing single arches at a time that the first arch could have pulled something wrong into the local cache) | 15:25 |
| mordred | but that's not the situation | 15:25 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Run Zuul as the zuuld user https://review.opendev.org/726958 | 15:34 |
| *** ykarel is now known as ykarel|away | 15:38 | |
| *** ysandeep is now known as ysandeep|afk | 15:45 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Better pre merge testing of mirror configs https://review.opendev.org/729305 | 15:47 |
| clarkb | I've discovered that our quay caching mirror isn't working via ^. I'm also not entirely sure that the red hat registry is working properly either | 15:49 |
| clarkb | sshnaidm: ^ I think both were added for tripleo, any idea if they are being used? | 15:50 |
| clarkb | I see the fix for quay (or at least the first thing to fix) which I'll try to get up soon | 15:50 |
| sshnaidm | clarkb, looking | 15:52 |
| corvus | clarkb: is https://mirror01.dfw.rax.opendev.org:4444/ supposed to be reachable? | 15:53 |
| corvus | clarkb: (i'm just trying out the various urls in that change) | 15:54 |
| mordred | clarkb: I believe there have been many quay issues recently | 15:54 |
| clarkb | corvus: not yet, https://review.opendev.org/#/c/728986/1 adds it (but its also buggy related to the quay thing) | 15:54 |
| mordred | clarkb: folks in #ansible-devel were talking a few hours ago about quay being broken | 15:54 |
| clarkb | I'm about to squash the two changes together since the additional testing is useful | 15:54 |
| corvus | clarkb: ah, the 8081 is the non-ssl version of that? | 15:55 |
| clarkb | corvus: yes | 15:55 |
| clarkb | increments start at 8080/4443 and bump by one for each case from there | 15:55 |
| corvus | clarkb: got it. i'll see if i can find a better query for that (your TODO) | 15:55 |
| clarkb | corvus: thanks | 15:55 |
| *** cloudnull has joined #opendev | 15:56 | |
| cloudnull | o/ | 15:56 |
| sshnaidm | clarkb, as cloudnull points we use quay in some of our jobs | 15:57 |
| sshnaidm | clarkb, so, it's for tripleo, right | 15:57 |
| * mordred waves to cloudnull | 15:58 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Enable ssl on all mirror vhosts https://review.opendev.org/728986 | 15:59 |
| cloudnull | o/ https://media0.giphy.com/media/IgGLggVL4HXYDAot0Y/200.gif | 15:59 |
| clarkb | corvus: ^ that is the squash I'll abandon the old child now | 15:59 |
| clarkb | cloudnull: sshnaidm so what I discovered is nothing on our mirrors is listening on the quay proxy port | 16:00 |
| clarkb | so it isn't possible for that to work at all as I understand it | 16:00 |
| cloudnull | I think I added quay, as well as rh registry, to the reverse proxy config a while back. | 16:01 |
| cloudnull | though its quite likely i did it wrong | 16:02 |
| clarkb | ya thats why I was asking if it is used anywhere | 16:02 |
| clarkb | (because I think we are about to change it in an effort to make it work but that may cause fallout on your side?) | 16:02 |
| *** lpetrut has joined #opendev | 16:02 | |
| cloudnull | we intend to use quay instead of docker.io but that transition has been slow moving | 16:02 |
| cloudnull | well quay is having a bad day today (throwing 500s) so today would be a good time to change it :D | 16:03 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add DaemonSet check for wait-for-pods role https://review.opendev.org/728503 | 16:03 |
| mordred | cloudnull: :) | 16:03 |
| cloudnull | its alraedy broken, cant be more broken | 16:03 |
| clarkb | cloudnull: k | 16:03 |
| cloudnull | if things change internally, we'll adjust. | 16:04 |
| cloudnull | clarkb I appreciate you fixing something i likely broke :D | 16:05 |
| clarkb | cloudnull: well ist a new feature that wasn't quite working | 16:05 |
| cloudnull | I'm someone who sees most bugs as just under appreciated features | 16:06 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 16:06 |
| clarkb | thats the first change to fix quay, its possible there will be more needed but we need that one at least I think | 16:06 |
| cloudnull | ++ | 16:07 |
| cloudnull | makes total sense | 16:07 |
| corvus | clarkb: you want to test that the version check returns okay, or should we list some tags on a repository? | 16:07 |
| * hrw out | 16:08 | |
| clarkb | corvus: I think we're ok with any normal expected output from the backend | 16:08 |
| clarkb | corvus: The idea would probably be to set up a docker client and pull through each of those, but for now simple is probably sufficient | 16:08 |
| clarkb | *the ideal | 16:08 |
| corvus | clarkb: heh, to be fair, the Forbidden is expected normal output too :) | 16:08 |
| clarkb | oh is it? | 16:08 |
| corvus | clarkb: yeah, a token is required for any access to docker.io (which may have an impact on how well we're actually mitigating docker outages) | 16:09 |
| clarkb | ah I see. We get the json doc back on the v2 side but just a http forbidden response for v1? | 16:09 |
| corvus | (i wonder if we're caching anything?) | 16:09 |
| clarkb | corvus: I'm pretty sure we're caching the images I seem to recall checking that at one point, but it is possible that is just a bw reduction and not reliability thing for us | 16:10 |
| corvus | clarkb: i think authentication is required for both versions? | 16:11 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 16:11 |
| clarkb | corvus: ya I think you're right | 16:11 |
| clarkb | unrelated on the other thing I've been pulling on https://zuul.opendev.org/t/openstack/build/5f6f67d2b83a441eaa5cb61aec6f5c37/log/jvb01.opendev.org/docker/jitsi-meet-docker_jvb_1.txt shows that the next thing for jvb scale out is configuring additional jvb's to talk to prosody somewhere other than localhost (and then I'll need to punch a firewall hole) | 16:12 |
| openstackgerrit | Rafael Folco proposed opendev/elastic-recheck master: DNM: Test query https://review.opendev.org/729319 | 16:15 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 16:16 |
| *** hashar has joined #opendev | 16:18 | |
| *** rpittau is now known as rpittau|afk | 16:22 | |
| zbr | what is the maintenance status of elastic-recheck repo? asking as it seems to need some maintenance. | 16:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning more puppet modules https://review.opendev.org/729321 | 16:25 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 16:30 |
| clarkb | zbr: I've been trying to encourage the qa team to use it more and be more involved since the process of managing queries is largely a qa activity | 16:31 |
| clarkb | zbr: what sort of maintenance are you talking about? | 16:31 |
| zbr | clarkb: multiple levels, both query maintenance and also code (failing to run tox locally). | 16:32 |
| zbr | i think that I have some people interested in my team, they will start proposing CRs. | 16:33 |
| clarkb | for query maintenance I think we really need to get the qa team involved. I don't think it will be sustainable otherwise (we aren't well positioned to debug why openstack is failing and so on) | 16:33 |
| clarkb | for tox I thinkwe can work on fixing that | 16:33 |
| zbr | sadly when they asked me for help, i realized that I need to fix few things. | 16:33 |
| zbr | yep, i do plan to advertise it to our qa team. | 16:34 |
| zbr | one remark they had was concern regarding the time to merge a new rule, as if takes a lot of time it may not prove practical. | 16:34 |
| zbr | anyway, I will propose few changes to fix testing jobs first. | 16:35 |
| clarkb | zbr: right the issue is we need the qa team to review changes | 16:35 |
| openstackgerrit | Merged zuul/zuul-jobs master: Don't require tox_envlist https://review.opendev.org/726829 | 16:35 |
| clarkb | historically mtreinish and mriedem have "owned" that review process. But they've both moved on to other things and no one has stepped in to fill the gap | 16:35 |
| clarkb | I don't think the infra teams are well positioned to say what is and isn't a good query for identifying bugs, that should fall to the qa team | 16:35 |
| clarkb | and if the qa team isn't interested in doing that we can see if anyone else is | 16:36 |
| clarkb | (one of the long standing todos with taht toolchange is to split the tooling away from the config then the interested debuggers can own the configs/queries and we can just run the service for them) | 16:36 |
| clarkb | unfortunately they are have been combined since day one which makes this confusing | 16:36 |
| zbr | there is also a considerable number of issues that are infra-related, for which queries are very useful | 16:36 |
| *** dpawlik has quit IRC | 16:37 | |
| clarkb | right this sin't to say infra related issues never cause problems. Its identifying that the job of debugging failures should fall to the qa team | 16:37 |
| zbr | in case position is open, i will be interested in nurturing it | 16:37 |
| clarkb | if they identify an issue on the infrastructure we tend to try and fix it | 16:37 |
| clarkb | and the reason for that is the infra team can't debug openstack and k8s and help and all the other technology in use | 16:38 |
| clarkb | the people working with those tools can | 16:38 |
| zbr | yep, but if jobs are failing due to more or less random mirror issue, we need to be able to see how often it happens, is not only about fixing it right away. | 16:39 |
| zbr | some stuff may prove flaky and it would be very important to be able to identify this (docker hub, centos repos, ....), so people do not waste time investigating a bug which is caused by "external forceds". | 16:40 |
| clarkb | I agree, but I still think we can't be considered primary debuggers | 16:41 |
| clarkb | add the bug for mirror is flaky, notify us about it, and we'll try to fix it | 16:41 |
| zbr | my guess is that if we make it easier to use, and people start seeing reports from it, maybe QA people will be more inclined to use it | 16:41 |
| *** ysandeep|afk is now known as ysandeep | 16:43 | |
| fungi | reports like http://status.openstack.org/elastic-recheck/ and http://status.openstack.org/openstack-health/ or something else? | 16:44 |
| fungi | problem is so many of those queries (especially supposed "infrastructure-related" ones) are tracking symptoms, not causes | 16:45 |
| clarkb | yup that top one is caused by dns issues when jobs break host dns, packages going python3 only and trying to install on python2, pypi outages and so on | 16:46 |
| fungi | for example, the top query there right now for "Pip fails to find distribution for package" is matching the string "No matching distribution found for" which can be caused by outages or by bugs in projects | 16:47 |
| fungi | yeah, what clarkb said | 16:47 |
| fungi | so tracking symptoms doesn't tell us much | 16:47 |
| *** lpetrut has quit IRC | 16:50 | |
| knikolla | is ubuntu one broken for logging in to gerrit? | 16:57 |
| corvus | i confirm it's malfunctioning | 16:58 |
| corvus | i end up at the generic gerrit openid page | 16:58 |
| corvus | and i just tried again and it worked | 16:59 |
| corvus | knikolla: try again? maybe it was a momentary outage of ubuntu one? | 16:59 |
| knikolla | corvus: tried again and works now | 16:59 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Bumped flake8 https://review.opendev.org/729328 | 17:01 |
| corvus | clarkb: i don't understand how effective caching can be happening due to the authorization requirement. at least looking at the first part of the process (the API getting tags, manifests, etc, before we get to blob serving), we have to pass an authorization header in, and there's no cache-control header returned. so i assume apache would at least treat each request differently, because each one is going | 17:02 |
| corvus | to arrive with a different authz header. | 17:02 |
| corvus | clarkb: (we could consider setting CacheQuickHandler which may bypass the authz check, which may be fine for this situation -- but i don't think we're doing that right now) | 17:03 |
| corvus | clarkb: i haven't done manual requests all the way down to the level of the blob serving, so maybe we're caching the blobs themselves effectively, just not the api? | 17:04 |
| clarkb | corvus: I believe we tell apache to ignore that stuff | 17:04 |
| clarkb | corvus: specifically we tell apache it is safe to cache sha256 addressed items iirc because those are not going to change | 17:05 |
| clarkb | so ya it could be that it is just the blob content and not the api | 17:05 |
| corvus | sounds like it | 17:05 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Bumped flake8 https://review.opendev.org/729328 | 17:06 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Add py38 test jobs https://review.opendev.org/729330 | 17:06 |
| corvus | clarkb: oh wait, apparently cachequickhandler is *on* by default | 17:06 |
| corvus | clarkb: so, erm, i don't know why the authorization header is still required | 17:07 |
| corvus | (but it does appear to be) | 17:07 |
| corvus | aha | 17:07 |
| corvus | Requests with an "Authorization" header (for example, HTTP Basic Authentication) are neither cacheable nor served from the cache when mod_cache is running in this phase. | 17:07 |
| corvus | so that applies to us | 17:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Enable ssl on all mirror vhosts https://review.opendev.org/728986 | 17:08 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 17:08 |
| clarkb | corvus: and the blob serviing is just out of a cdn without auth I think | 17:09 |
| clarkb | corvus: so that woudl explain it? | 17:09 |
| corvus | clarkb: yep | 17:09 |
| clarkb | also testing is good, I'm finding all sorts of bugs :) | 17:09 |
| zbr | fungi: clarkb: we will always have false-positive on matches, that not the problem. The value is in quantity, if we spot random isolated issues we can assume it was related to the change itself, but if you see peaks of lots of similar errors at the same time, that is when you realize that is likely an infra issue. | 17:11 |
| zbr | when I say infra, it does not mean infra managed by opendev, any kind of infra. | 17:12 |
| zbr | can we drop py27 support in elastic-recheck? | 17:12 |
| clarkb | zbr: thats just not true though. We see spikes due to bugs in the software all the time | 17:12 |
| clarkb | using the top bug of pip having issues and perfect example is when a python package switches to python3 only and all the python2 jobs try to install it | 17:13 |
| clarkb | you'll get a spike there | 17:13 |
| fungi | unless you want to consider the merged source code you're integrating as "infrastructure" | 17:13 |
| clarkb | has nothing to do with infrastructure but with buggy requirements lists | 17:13 |
| clarkb | zbr: yes I think we can drop py27 support in e-r | 17:13 |
| zbr | hurrah! so happy each time I ditch a py27 env. | 17:14 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Add py38 test jobs https://review.opendev.org/729330 | 17:14 |
| corvus | i've seen a few system-config-run jobs fail today due to an ubuntu keyserver timeout | 17:15 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 17:17 |
| clarkb | corvus: that'll be jobs that use ppa's I think | 17:17 |
| corvus | yes, buth for ze01.openstack.org | 17:17 |
| clarkb | corvus: one way to address that is to vendor the gpg pubkey for the ppa | 17:17 |
| clarkb | corvus: I believe the zuul docker images do the vendoring now for this reason, doing similar in the anible puppet probably a good idea | 17:17 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/elastic-recheck master: Drop py27 and add py38 jobs https://review.opendev.org/729330 | 17:18 |
| corvus | mordred: ^ reckon we ought to do that? | 17:18 |
| corvus | mordred: vendor ppa keys | 17:18 |
| mordred | corvus: yeah | 17:19 |
| mordred | I think fetching the keys from the keyservers has proven to be unnecessarily flaky vs. putting the key into git - we refer to them by id anyway, so if someone changes the key we still have to change what's in git | 17:20 |
| corvus | okay, i'll take a look at what we do in docker and see if i can adapt it to ansible | 17:20 |
| mordred | corvus: I think we have some places in ansible where we're vendoring already | 17:20 |
| corvus | mordred: cool any hints? | 17:21 |
| mordred | corvus: playbooks/roles/install-docker has one version | 17:21 |
| clarkb | zbr: we may need to switch where we run it to use python3 but python3 is available on status.openstack.org | 17:21 |
| mordred | corvus: which is probably a fine way to do it for production ansible | 17:21 |
| corvus | mordred: ack, thanks | 17:22 |
| mordred | corvus: there is an "easier" version we're using in docker images where we just drop a file in a dir - but I think it needs newer apt to work - so I think what's in install-docker is the right way to go | 17:22 |
| zbr | clarkb: ok. | 17:22 |
| mordred | corvus: also playbooks/roles/zuul-executor/tasks/main.yaml :) | 17:23 |
| corvus | mordred: wow that's some cognitive dissonance there :) | 17:25 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: WIP: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 17:28 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Add storyboard project for ansible-role-lunasa-hsm https://review.opendev.org/729334 | 17:35 |
| zbr | clarkb: do i have to update puppet-elastic_recheck or do we have exiting ansible for it? | 17:37 |
| clarkb | zbr: its still puppet | 17:37 |
| zbr | clarkb: ouch, my puppeteering skills are as low as possible. what version do we use? | 17:40 |
| mordred | 4 | 17:41 |
| clarkb | the puppet version shouldn't matter too much. I expect its just a matter of changing pip to pip3 | 17:43 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed opendev/puppet-elastic_recheck master: WIP: Use py3 with elastic-recheck https://review.opendev.org/729336 | 17:43 |
| zbr | https://260e67f8ff0e2abb7e19-6ad58f996ee59ccfe54ab476a81112a3.ssl.cf2.rackcdn.com/729336/1/check/legacy-puppet-lint/e13791f/job-output.txt | 17:53 |
| zbr | apparently being trolled | 17:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 17:55 |
| clarkb | zbr: ya that was a linter update we've been addressing as we fix things. The update itself is just silly but whatever puppet is gonna puppet and it will be gone one day | 17:55 |
| clarkb | zbr: have to remove the :: prefix beacuse it isn't necessary anymore (its actually perfectly valid still but they want you to remove it) | 17:56 |
| clarkb | I need to pop out now for some late breakfast/early lunch before the meeting | 17:57 |
| zbr | i am clueless, as I unable to even get something useful from running `rake`.LoadError: cannot load such file -- puppetlabs_spec_helper/rake_tasks | 17:58 |
| clarkb | zbr: manifests/bot.pp - WARNING: class included by absolute name (::$class) on line 32 and manifests/init.pp - WARNING: class included by absolute name (::$class) on line 42 immediately above the ERROR line are the problem | 17:59 |
| clarkb | zbr: if you'd like I can push a fix for it | 18:01 |
| clarkb | but may have to happen a bit later in the day as I've got a few things I need to do between now and the next 2 hours | 18:01 |
| zbr | it would be great, is late here and I am still trying to repair the ensure-docker, as is a blocker for tripleo test hobs. | 18:02 |
| zbr | clarkb: or at least comment with hints if you can, no pressure on that one. but hints are highly appreciated | 18:02 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 18:07 |
| *** ysandeep is now known as ysandeep|away | 18:13 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 18:19 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv https://review.opendev.org/726830 | 18:23 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove unused tox_envlist in fetch-subunit-output https://review.opendev.org/729348 | 18:24 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Deprecate default tox_envlist: venv https://review.opendev.org/726830 | 18:35 |
| *** chandankumar is now known as raukadah | 18:48 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Enable ssl on all mirror vhosts https://review.opendev.org/728986 | 18:52 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 18:52 |
| clarkb | I think ^ may actually have a chance of passig now | 18:52 |
| *** diablo_rojo has joined #opendev | 18:52 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 18:54 |
| clarkb | and ^ is getting closer to where the firewall will matter | 18:55 |
| * clarkb preps for meeting now | 18:55 | |
| *** roman_g has quit IRC | 18:59 | |
| *** hashar has quit IRC | 18:59 | |
| *** roman_g has joined #opendev | 19:05 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: add simple test runner https://review.opendev.org/728684 | 19:09 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: add simple test runner https://review.opendev.org/728684 | 19:11 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Add storyboard project for ansible-role-lunasa-hsm https://review.opendev.org/729334 | 19:12 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Use ensure-nodejs in Gerrit deployment testing https://review.opendev.org/729362 | 19:18 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Update OpenDev Manual URL in new contributor intro https://review.opendev.org/728479 | 19:30 |
| openstackgerrit | Jeremy Stanley proposed opendev/jeepyb master: Update OpenDev Manual URL in new contributor intro https://review.opendev.org/728479 | 19:31 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Configure ansible-role-lunasa-hsm for release https://review.opendev.org/729334 | 19:31 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-docker: workaround for centos-8 conflicts https://review.opendev.org/703053 | 19:55 |
| corvus | mordred: i think you're right, we didn't decide one way or the other. i *think* the use-ips-from-inventory thing is going well enough we don't need /etc/hosts, but maybe it's still a good idea? | 19:58 |
| mordred | corvus: yeah - I think using ips from inventory is the better idea in most of the cases | 19:58 |
| corvus | mordred: ok; we'll see how that goes, and we've got 726910 in our back-pocket if we need it | 19:59 |
| mordred | but I could imagine things that maybe want to be configured by hostname expecting to be able to talk to each other or something? | 19:59 |
| mordred | yeah | 19:59 |
| corvus | mordred: mostly it's just getting that change past the ppa gauntlet right now | 19:59 |
| mordred | turns out it's a very easy patch :) | 19:59 |
| clarkb | mordred: corvus: not sure if this changes your opinions there but the use case I've got is I need to tell jvb servers to talk to the meetpad prosody server | 20:00 |
| clarkb | I can look up the ip via inventory or I can also just say talk to meetpad.opendev.org port 5222 | 20:01 |
| fungi | heads up, if nobody's noticed, our gerrit deployment test jobs for versions other than 2.13 have been broken by the ensure-nodejs transition in zuul-jobs, https://review.opendev.org/729362 fixes them | 20:01 |
| corvus | clarkb, mordred: that sounds like a straightforward use-case for /etc/hosts | 20:03 |
| ianw | fungi: you might be interested in https://review.opendev.org/728743 to generate the ssl cert check list automatically. yesterday i got stuck on testing ... we need some way in testinfra to know what job we're running under | 20:04 |
| corvus | clarkb, fungi, mordred: i restored and +2d https://review.opendev.org/726910 | 20:04 |
| mordred | corvus: ++ agree | 20:04 |
| corvus | ianw: why should we care what job we're running? | 20:05 |
| ianw | corvus: i that case, i want to deploy the ssl check on bridge.openstack.org to avoid a whole other node. so i want to check in the output configuration that it put the letsencrypt hosts into the config file ... but what nodes go in the list varies by job | 20:07 |
| clarkb | mordred: corvus I've +2'd it | 20:07 |
| clarkb | er +3'd it even | 20:07 |
| fungi | as have i | 20:07 |
| corvus | ianw: how about specifying the expected list of certs as a job variable? | 20:09 |
| ianw | corvus: yeah the bit i need to work on is getting that into testinfra as something usable | 20:10 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: WIP add support for multiple jvbs behind meetpad https://review.opendev.org/729008 | 20:10 |
| corvus | ianw: maybe write it out to a file that we read in from testinfra | 20:10 |
| clarkb | ok ^ has been rebased on the firewall change and I'll just recheck it once the /etc/hosts change lands | 20:10 |
| corvus | ianw: (write it to a file in ansible then read in in the testinfra python) | 20:10 |
| ianw | corvus: yep, or we can set environment variables. anyway, i haven't looked at it yet, i only realised what was wrong last night :) | 20:11 |
| ianw | i'll do something generic-ish and then base the change on that | 20:11 |
| clarkb | corvus: etherpad updated to address your ???s thanks | 20:11 |
| ianw | but, i think autogenerating the ssl domain list for checking has legs, that bit all seems to work | 20:12 |
| corvus | clarkb: http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-03-25.log.html#t2020-03-25T14:08:45 | 20:12 |
| corvus | clarkb: that's frickler's scaling link | 20:12 |
| corvus | couple of interesting links there | 20:13 |
| clarkb | thanks | 20:13 |
| clarkb | corvus: that is a lot of jvb servers. Now to see if I can find how large they are | 20:14 |
| corvus | clarkb: i think there are some things in https://ffmuc.net/wiki/doku.php?id=knb:meet-server#prosody_setup_for_loadbalancing_control_server that aren't in your patch | 20:16 |
| corvus | clarkb: the prosody config? | 20:16 |
| clarkb | corvus: ya I'm looking over it now. Some things we already seem to have like prosody listening on 0.0.0.0 | 20:16 |
| clarkb | and for the component secrets I thought I read that with the Multi User Chat "MUC" setup that isn't necessary anymore | 20:17 |
| clarkb | the stuff about stats looks important for load balancing though | 20:17 |
| clarkb | https://github.com/jitsi/docker-jitsi-meet/blob/master/jvb/rootfs/defaults/sip-communicator.properties is the config we are toggling via the docker-compose .env file. That apepars to already use stats | 20:19 |
| corvus | clarkb: what do you think about the admin section? | 20:19 |
| clarkb | corvus: that admins bit is missing from our prosody config so that may need to be added | 20:20 |
| clarkb | fwiw the jvb logs in the job have been helpful in figuring out what is missing so far, I'm hoping that will give us an indication if there are more things to change | 20:20 |
| openstackgerrit | Clark Boylan proposed opendev/puppet-elastic_recheck master: Use py3 with elastic-recheck https://review.opendev.org/729336 | 20:24 |
| clarkb | zbr: ^ I think that will be happier with linting | 20:24 |
| clarkb | infra-root I can monitor https://review.opendev.org/#/c/728986/4 this afternoon if I can get a second review on it (thanks ianw for rereviewing it) | 20:27 |
| clarkb | thats the https on more mirror vhosts | 20:27 |
| clarkb | and now to review the firewall change | 20:29 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add OpenEdge CI mirror to Cacti config https://review.opendev.org/727389 | 20:33 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add missing HTTPS ports in ssldomains file https://review.opendev.org/727418 | 20:33 |
| openstackgerrit | Merged opendev/system-config master: Run multi-node-hosts-file in run-base-pre https://review.opendev.org/726910 | 20:34 |
| mordred | ianw: replied on https://review.opendev.org/#/c/720892 - I did a followup to remove more things and have discovered that we are in fact using puppet-openafs on mirror-update | 20:38 |
| ianw | oh yeah ... that's in the todo list! | 20:39 |
| mordred | ianw: I think we can just remove the openstack_project::server afs block | 20:40 |
| ianw | if it matches the afs-client group i guess so | 20:41 |
| mordred | looks like it does | 20:42 |
| ianw | mirror-update[0-9]*.opendev.org | 20:42 |
| ianw | i think that might what to go to open* ? | 20:42 |
| clarkb | corvus: why do we split iptables_allowed_groups into iptables_base_allowed_groups and iptables_extra_allowed_groups? Everything else about teh chagne lgtm but that jumped out to me | 20:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Stop cloning more puppet modules https://review.opendev.org/729321 | 20:45 |
| mordred | clarkb: I think base / extra allows us to have a global allowed groups and then to just add additional ones per service | 20:46 |
| mordred | clarkb: otherwise if we wanted to add a service we'd have to remember to copy in any global groups to it | 20:46 |
| clarkb | gotcha | 20:46 |
| mordred | (we do that pattern in some other places already) | 20:46 |
| corvus | yes that's it | 20:51 |
| corvus | in practice, that's going to be hard to use because in order for it to appear in the rules in a test job, some node in the group is going to have to be in the inventory. which is weird for a supposedly 'global' group. that's part of why cacti isn't in there right now. but i thought it worth keeping the pattern. | 20:52 |
| corvus | clarkb, mordred: note also the dependency on https://review.opendev.org/728952 and its parent | 20:53 |
| clarkb | I had missed that, looking now | 20:56 |
| clarkb | corvus: that for loop with the split is a neat trick | 21:00 |
| clarkb | I've approved the zuul-jobs stack | 21:01 |
| corvus | clarkb: python is still occasionally fun :) | 21:06 |
| corvus | clarkb: any day i can use a for/else loop is a good day | 21:06 |
| clarkb | I've approved the mirror ssl'ing change | 21:07 |
| clarkb | I'll be around to watch it | 21:07 |
| openstackgerrit | Merged zuul/zuul-jobs master: Update flake8 ignore rules to match Zuul https://review.opendev.org/729010 | 21:09 |
| openstackgerrit | Merged zuul/zuul-jobs master: Allow mapping additional hostvars in write-inventory https://review.opendev.org/728952 | 21:15 |
| openstackgerrit | Merged opendev/system-config master: Add OpenEdge CI mirror to Cacti config https://review.opendev.org/727389 | 21:24 |
| openstackgerrit | Merged opendev/system-config master: Add missing HTTPS ports in ssldomains file https://review.opendev.org/727418 | 21:28 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 21:28 |
| corvus | mordred: ^ that look like what you were thinkin? | 21:28 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: DNM: sanity check https://review.opendev.org/729402 | 21:30 |
| clarkb | corvus: I think that may only work on bionic and newer? xenial needs the old apt add-key command or whatever it was | 21:32 |
| mordred | corvus: yes! you might want to put playbooks/roles/install-apt-repo into the files: list in zuul.d in places where zuul-executor role is listed | 21:32 |
| clarkb | fungi: ^ you probably know off the top of your head | 21:32 |
| mordred | clarkb: I think the ansible module should take care of that | 21:32 |
| clarkb | oh aha | 21:32 |
| mordred | (I was going to mkae the same comment - but this is using apt_key - not just putting the file in place) | 21:32 |
| clarkb | roger | 21:32 |
| mordred | clarkb: we'll find out! | 21:33 |
| clarkb | being able to test so much of everything before we merge stuff has been excellent | 21:33 |
| corvus | ah yeah, i'll add the zuul conf | 21:33 |
| *** calcmandan has quit IRC | 21:34 | |
| clarkb | I've rechecked the jvb change which should use the /etc/hosts magic now | 21:34 |
| clarkb | and that should hopefully tell us more useful things | 21:34 |
| *** calcmandan has joined #opendev | 21:35 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 21:36 |
| openstackgerrit | Merged opendev/system-config master: Enable ssl on all mirror vhosts https://review.opendev.org/728986 | 21:38 |
| fungi | clarkb: yes, ubuntu-bionic is probably new enough, debian-buster and ubuntu-focal definitely are, ubuntu-xenial and debian-stretch are not | 21:38 |
| * fungi checks manpage | 21:38 | |
| fungi | yeah, bionic is new enough | 21:40 |
| *** slaweq has quit IRC | 21:41 | |
| fungi | the apt-key(8) manpage on a bionic server mentions: Instead of using this command a keyring should be placed directly in the /etc/apt/trusted.gpg.d/ directory with a descriptive name and either "gpg" or "asc" as file extension. | 21:46 |
| fungi | what it doesn't exactly say is that you should use .gpg on raw exported keyring files and .asc on pem-format keys | 21:49 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option to prefer https/ssl in configure-mirrors https://review.opendev.org/729407 | 21:58 |
| hrw | fungi: good to know ;) | 22:05 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Escape use of % in tox.ini to avoid interpolation https://review.opendev.org/729412 | 22:16 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 22:17 |
| clarkb | infra-root config-core 729412 addresses a problem that the new tox role reorg exposes in that tox.ini. Assuming it works (eg the escape doesn't break the print there) then we'll want to land that | 22:20 |
| johnsom | Is IPv6 not working at openedge a known issue or something I need to look into? | 22:20 |
| clarkb | johnsom: that is not a known issue, its an ipv6 "only" cloud so we should expect it to work | 22:20 |
| clarkb | johnsom: do you hvae examples? | 22:20 |
| fungi | openedge is natively v6 and does all public v4 via nat, yeah | 22:20 |
| johnsom | https://89169a9b326532519e5e-7f1ab0a954aa845ce901e1b3383eb285.ssl.cf1.rackcdn.com/729391/1/check/octavia-v2-dsvm-scenario/b6e849e/ | 22:21 |
| johnsom | Just the Ipv6 tests failed | 22:21 |
| clarkb | oh thats in the nested cloud | 22:21 |
| clarkb | I don't think that relies on any of the cloud side networking in openedge | 22:21 |
| johnsom | Well, we have tempest picking the "external" ipv6 net so resources are reachable from the tempest instance. So, we do hop out typically. | 22:23 |
| clarkb | you shouldn't hop out | 22:23 |
| johnsom | I just haven't noticed those before, so wondered if it was still coming up. | 22:23 |
| clarkb | on half the clouds there isn't anything to hop out to | 22:23 |
| johnsom | Well, yeah, I guess we don't really. I'm thinking of "hop out" of neutron's config. so not really going out. | 22:24 |
| johnsom | I will dig through logs and see what I find. Thanks | 22:24 |
| clarkb | ah ya. I know for ipv4 we explicitly create an interface on the fip network so that the route table sees it as locally attached | 22:24 |
| clarkb | you may need something like that for ipv6? | 22:24 |
| johnsom | We haven't for a few years now, so wouldn't expect that to change. | 22:25 |
| clarkb | also I don't see the worlddump log whihc would probably be helpful in this case | 22:25 |
| clarkb | I guess because worlddump is only when devstack fails | 22:25 |
| clarkb | not when tempest fails | 22:25 |
| clarkb | (changing that to be when the job is going to fail would probably be useful) | 22:26 |
| johnsom | Ha, yeah, I asked about that a few weeks ago. grin great minds think alike eh? | 22:26 |
| ianw | these days it's probably better as a post role | 22:38 |
| clarkb | fwiw I've discovered that I forgot to add holes in the firewall for the mirror update. Everything else seems mostly happy. I'm working on a change to test via hitting the public interface so that we test that better in the future | 22:43 |
| johnsom | clarkb It looks like the IPv6 is working fine there. I see good addresses and VIPs. We can pass traffic to some other instances. So likely something in the nova or neutron level. | 22:44 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Open mirror ssl ports externally https://review.opendev.org/729416 | 22:50 |
| clarkb | ianw: ^ fyi | 22:50 |
| clarkb | I'll rebase the quay change on top of that | 22:50 |
| ianw | lgtm | 22:50 |
| clarkb | ianw: do you have a moment for https://review.opendev.org/#/c/729412/ whihc addresses a tox issue that zuul-jobs update exposed | 22:51 |
| ianw | also lgtm :) | 22:53 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 22:58 |
| clarkb | ianw: also I think we need to restart apache2 on all of the mirrors which our current setupt doesn't do (it just reloads to be as graceful as possible) | 22:59 |
| clarkb | I'll probably wait for the iptables updates to get in then I can do a global restart | 22:59 |
| ianw | oh, i thought that would create the new vhosts? maybe not as you say | 23:00 |
| clarkb | ianw: when I first figured out hwat was wrong on mirror.dfw at least I had to restart it then only localhost worked which was why I realized it was iptables at fault | 23:00 |
| clarkb | I checked netstat -lnp output to confirm before restarting apache2 | 23:01 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 23:07 |
| openstackgerrit | Merged openstack/project-config master: Escape use of % in tox.ini to avoid interpolation https://review.opendev.org/729412 | 23:09 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Vendor the apt repo gpg keys used for Zuul https://review.opendev.org/729401 | 23:12 |
| *** DSpider has quit IRC | 23:21 | |
| corvus | mordred: where did the value for gearman_server_ssl_key come from in commit f0b77485ec559aa9cde2a5066ecb72a2ffa47ea9 ? | 23:38 |
| corvus | the next error with testing zuul in the gate seems to be that the key does not match the cert | 23:38 |
| mordred | corvus: I probably just made one using openssl | 23:41 |
| corvus | mordred: ok, there are a lot of certs; maybe the wrong one got added :) | 23:41 |
| mordred | corvus: I imagine this is now one of those cases where we we'll need to override the public value too, since we'll want them to match | 23:42 |
| clarkb | hrm testinfra doesn't work quite the way I expected it to | 23:42 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 23:42 |
| corvus | mordred: oh! i wonder if that's what's going on -- a fake private key but only the real public key | 23:43 |
| mordred | corvus: like - isn't the cert in the public host_vars - but matches the private key | 23:43 |
| mordred | yeah | 23:43 |
| mordred | we need a fake public key too | 23:43 |
| corvus | ok. so yeah, i'll make a new fake pair then | 23:43 |
| mordred | corvus: there might be another issue ... I don't think our fake variables override our public hostvars | 23:43 |
| corvus | (i'd use zk-ca.sh for this, but this whole stack is in service of the patch that adds zk-ca.sh, so i'll just do it the hard way till we get things working) | 23:43 |
| mordred | (I think I ran in to that before but was able to punt on it) | 23:44 |
| mordred | but in any case - I believe that is the situation | 23:44 |
| corvus | mordred: ok. we can put all the certs in private vars. | 23:44 |
| corvus | that's too big of a change now, so i'll pick this up tomorrow | 23:45 |
| mordred | yeah. it's also possible I'm not right about that | 23:45 |
| mordred | OR | 23:45 |
| mordred | that maybe we should consider var precedence so that test fake vars _do_ take precedence over the public vars - no clue if it's even possible for us to do that | 23:45 |
| corvus | mordred: i don't see a reason to have the cert file as a public var | 23:46 |
| mordred | me either | 23:46 |
| mordred | mostly just thinking out loud | 23:46 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Open mirror ssl ports externally https://review.opendev.org/729416 | 23:48 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Listen on Quay Registry Mirror Ports https://review.opendev.org/729315 | 23:48 |
| clarkb | I'm not convinced ^ is entirely correct. I expected testinfra to make it easier to get the ip addrs of the host being operated on | 23:48 |
| mordred | clarkb: there you go expecting things again | 23:49 |
| clarkb | mordred: well even from the docs they are like "here is the addr object with the ip_addresses property" | 23:49 |
| clarkb | but then I rtfs'd and realized that the addr object is a class object that you need to instantiate | 23:49 |
| clarkb | its really there to do ip lookups of arbitrary names from the host | 23:49 |
| clarkb | so I'll use that to have the host ask its own name :/ | 23:50 |
| ianw | clarkb: or ... we pass the inventory in as a fixture which is what i'm coming at with 729418? | 23:50 |
| clarkb | ianw: oh ya that would work too | 23:51 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] testinfra: create a fixture of data from zuul https://review.opendev.org/729418 | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
