| *** slaweq has quit IRC | 00:01 | |
| clarkb | fungi: did we ever put inap back into service with a bigger mirror? | 00:01 |
|---|---|---|
| *** ociuhandu has quit IRC | 00:02 | |
| *** jklare has joined #openstack-infra | 00:02 | |
| ianw | i feel like i made that mirror? | 00:04 |
| clarkb | ianw: I guess the issue was we ran out of disk for our afs + apache caches | 00:05 |
| clarkb | the old server had had a cinder volume maybe or a bigger root disk? | 00:05 |
| clarkb | (I was sick at the tiem of this happening so memory may be fuzzy) | 00:05 |
| clarkb | I've confirmed that certcheck is now happy with the certs on opendev.org too | 00:09 |
| clarkb | emails should stop now | 00:09 |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 00:11 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix typo in script name https://review.opendev.org/695117 | 00:11 |
| ianw | clarkb: hrm, i don't remember than ... | 00:12 |
| *** xek has quit IRC | 00:12 | |
| ianw | https://review.opendev.org/#/c/695099/ passed, so i'd say ansible 2.9.1 on bridge is *probably* ok | 00:13 |
| *** apetrich has quit IRC | 00:15 | |
| *** slaweq has quit IRC | 00:18 | |
| *** armax has joined #openstack-infra | 00:25 | |
| *** slaweq has joined #openstack-infra | 00:42 | |
| *** rlandy|rover has quit IRC | 00:44 | |
| *** slaweq has quit IRC | 00:46 | |
| *** rfolco has quit IRC | 00:52 | |
| *** slaweq has joined #openstack-infra | 01:01 | |
| clarkb | ianw: fungi inap is still disabled according to project-config content. Looking at server flavors on inap an easy option would be to rebuild the server with flavor A1.16. This has 320GB of disk. Another option is a cinder volume, our quota does allow us 20 volumes and plenty of GB so we'd be able to do that in inap | 01:01 |
| clarkb | I can give that a go tomorrow morning likely | 01:01 |
| ianw | ++ | 01:02 |
| clarkb | let me know if either of you have a preference for appraoch above. Maybe benj_ has thoughts too? benj_ basically our existing cloud local mirror host doesn't have enough disk for its caches. Our two options are to supplement existing server with a cinder volume or build a new server with bigger root disk. | 01:02 |
| openstackgerrit | Ghanshyam Mann proposed openstack/openstack-zuul-jobs master: Explicit mention py2 for grenade py2 jobs https://review.opendev.org/695124 | 01:03 |
| *** slaweq has quit IRC | 01:05 | |
| *** goldyfruit_ has joined #openstack-infra | 01:07 | |
| fungi | clarkb: ahh, yeah, i was thinking of just attaching a 200gb cinder volume like we've done on other mirrors | 01:16 |
| fungi | carve into 2x 100gb logical volumes and mount to the two cache dirs | 01:16 |
| fungi | ianw: yeah, i confirmed from old graphs the previous inap mirror had separate volumes mounted on the cachedirs, the new one only had a rootfs and not nearly enough space | 01:17 |
| ianw | fungi: hrm, according to my notes i brought it up on 10-24 ... ? i feel like i would have added a volume but maybe i forgot :/ | 01:19 |
| fungi | or created the volumes but didn't add them to fstab? i haven't checked the cinder inventory there | 01:20 |
| clarkb | /dev/disk/by-uuid only shows two devices | 01:21 |
| clarkb | volume list doesn't show any volumes. I can create and attach, add to lvm and format, etc tomorrow | 01:21 |
| *** ociuhandu has joined #openstack-infra | 01:22 | |
| ianw | ok can see if i can take a look too after getting through some other bits | 01:22 |
| clarkb | k I'll check scrollback then | 01:23 |
| *** slaweq has joined #openstack-infra | 01:25 | |
| *** slaweq has quit IRC | 01:30 | |
| *** ociuhandu has quit IRC | 01:34 | |
| *** lifeless has quit IRC | 01:34 | |
| *** mriedem has quit IRC | 01:44 | |
| *** lifeless has joined #openstack-infra | 01:49 | |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 01:51 |
| openstackgerrit | Merged opendev/system-config master: vos-release: fix typo in script name https://review.opendev.org/695117 | 01:58 |
| *** goldyfruit_ has quit IRC | 01:58 | |
| *** jklare has quit IRC | 02:00 | |
| *** jklare has joined #openstack-infra | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:06 | |
| *** lifeless has quit IRC | 02:12 | |
| *** michael-beaver has quit IRC | 02:12 | |
| dmsimard | ianw: almost there -- it's installed and is enabled but fails during the ara-report role: http://paste.openstack.org/show/786377/ | 02:12 |
| *** slaweq has joined #openstack-infra | 02:13 | |
| ianw | ok, it's not writing on the db? | 02:16 |
| dmsimard | nope, my mistake for the database config | 02:16 |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 02:17 |
| *** igordc has quit IRC | 02:18 | |
| *** slaweq has quit IRC | 02:18 | |
| *** lifeless has joined #openstack-infra | 02:24 | |
| *** slaweq has joined #openstack-infra | 02:24 | |
| *** slaweq has quit IRC | 02:30 | |
| *** roman_g has quit IRC | 02:33 | |
| *** ociuhandu has joined #openstack-infra | 02:35 | |
| *** slaweq has joined #openstack-infra | 02:35 | |
| *** ociuhandu has quit IRC | 02:39 | |
| dmsimard | same error again... hrm | 02:43 |
| dmsimard | "rsync: link_stat \"/var/cache/ansible/ara.sqlite\" failed: No such file or directory (2)\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) | 02:44 |
| ianw | argh, key didn't roll out for local afs ... why ...? | 02:44 |
| dmsimard | I think it's because the configuration isn't being taken into account so it's at the default location instead of that one | 02:44 |
| *** slaweq has quit IRC | 02:48 | |
| *** gyee has quit IRC | 02:53 | |
| dmsimard | ohhhh, I know what the problem is :D | 02:53 |
| *** slaweq has joined #openstack-infra | 03:00 | |
| *** slaweq has quit IRC | 03:04 | |
| *** slaweq has joined #openstack-infra | 03:05 | |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 03:14 |
| *** slaweq has quit IRC | 03:16 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix key sourcing https://review.opendev.org/695129 | 03:23 |
| *** slaweq has joined #openstack-infra | 03:26 | |
| *** goldyfruit_ has joined #openstack-infra | 03:28 | |
| *** slaweq has quit IRC | 03:31 | |
| ianw | hrm, i think i found a problem; puppet overwrites the authorized_keys | 03:33 |
| *** slaweq has joined #openstack-infra | 03:35 | |
| *** slaweq has quit IRC | 03:40 | |
| *** ricolin has joined #openstack-infra | 03:43 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix key sourcing; disable exclusive key https://review.opendev.org/695129 | 03:44 |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 03:48 |
| *** slaweq has joined #openstack-infra | 03:55 | |
| *** slaweq has quit IRC | 04:02 | |
| *** liuyulong has quit IRC | 04:05 | |
| *** slaweq has joined #openstack-infra | 04:10 | |
| openstackgerrit | David Moreau Simard proposed opendev/system-config master: DNM: Test ara 1.2 for bridge https://review.opendev.org/695108 | 04:10 |
| *** slaweq has quit IRC | 04:15 | |
| *** armax has quit IRC | 04:18 | |
| *** slaweq has joined #openstack-infra | 04:39 | |
| *** slaweq has quit IRC | 04:44 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: use-buildset-registry: Vendor pytoml and remarshal https://review.opendev.org/695050 | 04:46 |
| *** goldyfruit_ has quit IRC | 05:00 | |
| *** slaweq has joined #openstack-infra | 05:01 | |
| *** rh-jelabarre has quit IRC | 05:07 | |
| *** slaweq has quit IRC | 05:08 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:20 | |
| *** soniya29 has joined #openstack-infra | 05:21 | |
| *** pcaruana has joined #openstack-infra | 05:25 | |
| *** Lucas_Gray has joined #openstack-infra | 05:28 | |
| *** slaweq has joined #openstack-infra | 05:29 | |
| *** iokiwi has quit IRC | 05:29 | |
| *** surpatil has joined #openstack-infra | 05:30 | |
| *** iokiwi has joined #openstack-infra | 05:30 | |
| *** ociuhandu has joined #openstack-infra | 05:31 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Dockerfile: add DEBUG environment flag https://review.opendev.org/694845 | 05:32 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon https://review.opendev.org/693464 | 05:32 |
| *** bdodd has quit IRC | 05:32 | |
| *** slaweq has quit IRC | 05:33 | |
| *** ociuhandu has quit IRC | 05:35 | |
| *** bdodd has joined #openstack-infra | 05:36 | |
| ianw | corvus: as discussed, didn't see any non-bugfix gitea items that I think would affect us on https://review.opendev.org/#/c/694894/ | 05:38 |
| *** bdodd has quit IRC | 05:41 | |
| *** slaweq has joined #openstack-infra | 05:42 | |
| *** Lucas_Gray has quit IRC | 05:45 | |
| *** bdodd has joined #openstack-infra | 05:46 | |
| *** slaweq has quit IRC | 05:47 | |
| openstackgerrit | Merged openstack/project-config master: Remove unused path variable from promote secrets https://review.opendev.org/694912 | 05:51 |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** pcrews has quit IRC | 06:14 | |
| *** slaweq has quit IRC | 06:15 | |
| *** pcrews has joined #openstack-infra | 06:20 | |
| *** SurajPatil has joined #openstack-infra | 06:26 | |
| *** slaweq has joined #openstack-infra | 06:27 | |
| *** surpatil has quit IRC | 06:28 | |
| *** slaweq has quit IRC | 06:32 | |
| *** larainema has joined #openstack-infra | 06:33 | |
| *** slaweq has joined #openstack-infra | 06:39 | |
| *** slaweq has quit IRC | 06:44 | |
| *** roman_g has joined #openstack-infra | 07:10 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** dpawlik has joined #openstack-infra | 07:14 | |
| *** slaweq has quit IRC | 07:15 | |
| *** pgaxatte has joined #openstack-infra | 07:19 | |
| *** udesale has joined #openstack-infra | 07:25 | |
| *** tosky has joined #openstack-infra | 07:35 | |
| *** priteau has quit IRC | 07:38 | |
| *** surpatil has joined #openstack-infra | 07:42 | |
| *** SurajPatil has quit IRC | 07:44 | |
| gmann | clarkb: fungi do you know why there is stable/train in master barbican grenade job inventory (any other grenade job is all master) - https://zuul.opendev.org/t/openstack/build/a3a17f8e4d054b6d9c417b44077159a1/log/zuul-info/inventory.yaml#32 | 07:53 |
| *** slaweq has joined #openstack-infra | 07:56 | |
| *** dpawlik has quit IRC | 07:56 | |
| *** ccamacho has joined #openstack-infra | 07:57 | |
| *** tesseract has joined #openstack-infra | 07:58 | |
| *** soniya29 has quit IRC | 08:04 | |
| *** roman_g has quit IRC | 08:06 | |
| *** ociuhandu has joined #openstack-infra | 08:09 | |
| *** tkajinam has quit IRC | 08:17 | |
| *** ociuhandu has quit IRC | 08:19 | |
| *** dpawlik has joined #openstack-infra | 08:21 | |
| *** dchen has quit IRC | 08:28 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** elod_off is now known as elod | 08:50 | |
| *** ralonsoh has joined #openstack-infra | 08:51 | |
| *** rpittau|afk is now known as rpittau | 08:53 | |
| *** ociuhandu has joined #openstack-infra | 08:55 | |
| *** ociuhandu has quit IRC | 08:59 | |
| *** soniya29 has joined #openstack-infra | 09:00 | |
| *** ociuhandu has joined #openstack-infra | 09:00 | |
| *** lucasagomes has joined #openstack-infra | 09:01 | |
| *** apetrich has joined #openstack-infra | 09:03 | |
| *** ociuhandu has quit IRC | 09:03 | |
| *** ociuhandu has joined #openstack-infra | 09:04 | |
| *** rfolco has joined #openstack-infra | 09:05 | |
| *** jamesmcarthur has quit IRC | 09:07 | |
| *** rfolco has quit IRC | 09:09 | |
| *** ociuhandu has quit IRC | 09:11 | |
| *** ociuhandu has joined #openstack-infra | 09:11 | |
| *** ociuhandu has quit IRC | 09:13 | |
| *** tosky has quit IRC | 09:13 | |
| *** SurajPatil has joined #openstack-infra | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:14 | |
| *** tosky has joined #openstack-infra | 09:14 | |
| *** SurajPatil is now known as SurajPatil|lunch | 09:15 | |
| *** ociuhandu has quit IRC | 09:15 | |
| *** jamesmcarthur has joined #openstack-infra | 09:15 | |
| *** ociuhandu has joined #openstack-infra | 09:16 | |
| *** surpatil has quit IRC | 09:16 | |
| *** ociuhandu has quit IRC | 09:17 | |
| *** ociuhandu has joined #openstack-infra | 09:18 | |
| *** ociuhandu has quit IRC | 09:19 | |
| *** ociuhandu has joined #openstack-infra | 09:20 | |
| *** Lucas_Gray has joined #openstack-infra | 09:22 | |
| *** ociuhandu has quit IRC | 09:26 | |
| *** ociuhandu has joined #openstack-infra | 09:27 | |
| *** ociuhandu has quit IRC | 09:29 | |
| *** ociuhandu has joined #openstack-infra | 09:29 | |
| *** pkopec has joined #openstack-infra | 09:31 | |
| *** ociuhandu has quit IRC | 09:32 | |
| *** ociuhandu has joined #openstack-infra | 09:32 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** ociuhandu has joined #openstack-infra | 09:35 | |
| *** ociuhandu has quit IRC | 09:37 | |
| *** iurygregory has joined #openstack-infra | 09:37 | |
| *** ociuhandu has joined #openstack-infra | 09:37 | |
| *** ociuhandu has quit IRC | 09:39 | |
| *** ociuhandu has joined #openstack-infra | 09:40 | |
| *** ociuhandu has quit IRC | 09:43 | |
| *** ociuhandu has joined #openstack-infra | 09:43 | |
| *** dpawlik has quit IRC | 09:45 | |
| *** ociuhandu has quit IRC | 09:45 | |
| *** ociuhandu has joined #openstack-infra | 09:46 | |
| *** rcernin has quit IRC | 09:46 | |
| *** Wryhder has joined #openstack-infra | 09:46 | |
| *** derekh has joined #openstack-infra | 09:47 | |
| *** ociuhandu has quit IRC | 09:48 | |
| *** Lucas_Gray has quit IRC | 09:48 | |
| *** ociuhandu has joined #openstack-infra | 09:49 | |
| *** Wryhder is now known as Lucas_Gray | 09:49 | |
| *** ociuhandu has quit IRC | 09:50 | |
| *** ociuhandu has joined #openstack-infra | 09:51 | |
| *** priteau has joined #openstack-infra | 09:52 | |
| *** ociuhandu has quit IRC | 09:52 | |
| *** ociuhandu has joined #openstack-infra | 09:53 | |
| *** gfidente has joined #openstack-infra | 09:55 | |
| *** ociuhandu has quit IRC | 09:56 | |
| *** ociuhandu has joined #openstack-infra | 09:57 | |
| *** jaosorior has joined #openstack-infra | 09:57 | |
| *** ociuhandu has quit IRC | 09:58 | |
| *** ociuhandu has joined #openstack-infra | 09:58 | |
| *** ruffian_sheep has joined #openstack-infra | 09:59 | |
| *** rascasoft has joined #openstack-infra | 10:00 | |
| *** priteau has quit IRC | 10:06 | |
| *** SurajPatil|lunch is now known as SurajPatil | 10:10 | |
| *** dpawlik has joined #openstack-infra | 10:20 | |
| *** dpawlik has quit IRC | 10:24 | |
| *** dtantsur|afk is now known as dtantsur | 10:27 | |
| AJaeger | gmann: I think the branch matcher at https://opendev.org/openstack/barbican/src/branch/stable/train/.zuul.yaml#L138 is the problem, those are evil. | 10:27 |
| AJaeger | So, https://review.opendev.org/689458 and friends might be needed | 10:27 |
| AJaeger | gmann: answered as well via email with the above | 10:30 |
| *** ricolin has quit IRC | 10:32 | |
| *** xek has joined #openstack-infra | 10:35 | |
| *** dtantsur is now known as dtantsur|brb | 10:48 | |
| *** dpawlik has joined #openstack-infra | 10:51 | |
| mordred | ianw: what's our current story for certs in openstack.org domains - that's not LE yet right? | 10:54 |
| frickler | mordred: iirc we can do LE there, too. just need the foundation folks to install a CNAME entry for the acme subdomain manually | 10:56 |
| mordred | ah - cool | 10:56 |
| * mordred needs a real cert on storyboard-dev to make forward progress on gerrit | 10:56 | |
| mordred | fungi: ^^ when you get up, could you put in an _acme_challenge for storyboard-dev? or do I need to ping jimmy? | 10:58 |
| mordred | uhm | 11:02 |
| mordred | we seem to be half-transitioned between storyboard.*.openstack.org and storyboard.*.opendev.org | 11:03 |
| mordred | the servers and the ansible and the puppet all seem to think they are .opendev.org - but there's no DNS entry | 11:03 |
| frickler | mordred: I haven't found any example of a openstack.org LE cert, though the spec mentions it. possibly migrating to storyboard-dev.opendev.org will actually be easier | 11:07 |
| mordred | yeah - since that should really just be adding dns records and adding a redirect | 11:08 |
| *** PrinzElvis has quit IRC | 11:08 | |
| *** dpawlik has quit IRC | 11:12 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10.0 https://review.opendev.org/690873 | 11:14 |
| *** ociuhandu has quit IRC | 11:21 | |
| *** lpetrut has joined #openstack-infra | 11:26 | |
| *** pgaxatte has quit IRC | 11:42 | |
| *** Lucas_Gray has quit IRC | 11:47 | |
| *** dpawlik has joined #openstack-infra | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:51 | |
| *** dpawlik has quit IRC | 11:53 | |
| *** hrw has joined #openstack-infra | 11:53 | |
| hrw | morning | 11:54 |
| *** rlandy has joined #openstack-infra | 11:55 | |
| *** rlandy is now known as rlandy|rover | 11:55 | |
| *** lmiccini has joined #openstack-infra | 11:55 | |
| hrw | can ubuntu mirror on mirror.sjc1.vexxhost.openstack.org machine get updated? it is out of sync breaking CI jobs | 11:55 |
| hrw | INFO:kolla.common.utils.kolla-toolbox:The following packages have unmet dependencies: | 11:55 |
| hrw | INFO:kolla.common.utils.kolla-toolbox: libssl-dev : Depends: libssl1.1 (= 1.1.1-1ubuntu2.1~18.04.4) but 1.1.1-1ubuntu2.1~18.04.5 is to be installed | 11:55 |
| hrw | https://046b6520e005cc815b3d-5ba6792c7427741c3f6f06d35721722e.ssl.cf5.rackcdn.com/695170/1/check/kolla-build-ubuntu-source/808dffa/kolla/build/000_FAILED_kolla-toolbox.txt.gz job log | 11:56 |
| frickler | hrw: we have an issue with afs volumes in general, not sure about the current state, but this is being handled already | 11:57 |
| hrw | frickler: thanks | 11:57 |
| *** SurajPatil has quit IRC | 12:08 | |
| *** PrinzElvis has joined #openstack-infra | 12:15 | |
| *** dave-mccowan has quit IRC | 12:17 | |
| *** larainema has quit IRC | 12:23 | |
| *** pcaruana has quit IRC | 12:24 | |
| *** ruffian_sheep has quit IRC | 12:29 | |
| *** jpena is now known as jpena|lunch | 12:29 | |
| *** soniya29 has quit IRC | 12:40 | |
| *** pgaxatte has joined #openstack-infra | 12:45 | |
| *** pcaruana has joined #openstack-infra | 12:50 | |
| *** derekh has quit IRC | 12:55 | |
| *** ociuhandu has quit IRC | 12:58 | |
| *** goldyfruit_ has joined #openstack-infra | 12:59 | |
| fungi | mordred: when frickler says "need the foundation folks to install a CNAME entry" what he really means is "add cnames in that domain in rackspace like we always have" | 12:59 |
| fungi | but sure, i can take a stab at it shortly | 12:59 |
| *** Lucas_Gray has joined #openstack-infra | 13:01 | |
| fungi | not realy half-transitioned on storyboard servers, we had simply declared a while back that we would start naming servers in the opendev.org domain if they were for services opendev would be providing, whether or not the service's domain was opendev.org at the time | 13:01 |
| *** derekh has joined #openstack-infra | 13:02 | |
| fungi | so that we didn't need to rebuild them later to change their instance names | 13:02 |
| fungi | mordred: but yeah, no reason i can see not to go ahead and set it up as storyboard-dev.opendev.org at this point | 13:04 |
| fungi | though i do have a to do item to reinstall it with python3 | 13:04 |
| fungi | that may wait until we dockerify the deployment | 13:04 |
| *** priteau has joined #openstack-infra | 13:05 | |
| fungi | it will break old storyboard-webcliennt draft builds until those expire out of the log archive, but we can always recheck after we update the jobs | 13:06 |
| *** ricolin has joined #openstack-infra | 13:07 | |
| fungi | as for the afs mirrors, the recovery from yesterdays hung server reboot looks like it will need to deal with stale vldb locks | 13:09 |
| fungi | Could not lock the VLDB entry for the volume 536870949. VLDB: vldb entry is already locked | 13:09 |
| fungi | errors like that | 13:09 |
| *** Lucas_Gray has quit IRC | 13:11 | |
| mordred | fungi: nod to all of the above | 13:11 |
| *** Lucas_Gray has joined #openstack-infra | 13:11 | |
| mordred | fungi: so - my primary motivation here is that storyboard-dev is pointed to by review-dev and so we inject the storyboard-dev ssl cert, which currently involves running a java command, and therefore would be something that would need to be done inside the contianer | 13:12 |
| mordred | HOWEVER | 13:12 |
| mordred | we don't do that on review.o.o - it's purely a step we do on review-dev ... so maybe just omitting it from review-dev and not blocking gerrit docker on the storyboard-dev chain (which we can also work in parallel) is fine? | 13:13 |
| mordred | (because the storyboard thread could quickly become _another_ fun project, albeit a worthwhile one) | 13:13 |
| fungi | oh, yep, that's purely to deal with the self-signed snakeoil cert on storyboard-dev | 13:13 |
| fungi | java doesn't like connecting to those without additional trust chain | 13:14 |
| *** goldyfruit_ has quit IRC | 13:16 | |
| mordred | yeah. which is where this all just gets gnarly :) | 13:17 |
| *** dave-mccowan has joined #openstack-infra | 13:21 | |
| *** iurygregory has quit IRC | 13:31 | |
| *** iurygregory has joined #openstack-infra | 13:32 | |
| *** mriedem has joined #openstack-infra | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:36 | |
| *** iurygregory has quit IRC | 13:36 | |
| *** iurygregory has joined #openstack-infra | 13:37 | |
| *** dpawlik has joined #openstack-infra | 13:39 | |
| mnasiadka | good afternoon | 13:41 |
| mnasiadka | Is there a way to add more logs to be pushed to logstash on a project level? I mean that list: https://opendev.org/opendev/base-jobs/src/branch/master/roles/submit-logstash-jobs/defaults/main.yaml#L9 | 13:42 |
| *** eharney has quit IRC | 13:43 | |
| *** ociuhandu has joined #openstack-infra | 13:44 | |
| *** roman_g has joined #openstack-infra | 13:48 | |
| *** ociuhandu has quit IRC | 13:48 | |
| *** liuyulong has joined #openstack-infra | 13:57 | |
| *** AJaeger has quit IRC | 13:58 | |
| fungi | mnasiadka: not as far as i know, but is there a reason not to just collect the log in question from any build which creates it? | 14:03 |
| mnasiadka | fungi: in kolla-ansible we put deploy logs in logs/{{ hostname }}/ansible/ - we would like to use the elastic-recheck functionality, but those logs are not in logstash - so we can't ;) | 14:04 |
| mnasiadka | fungi: other option is to add ansible/.* to the submit-logstash-jobs role I guess | 14:04 |
| *** AJaeger has joined #openstack-infra | 14:06 | |
| fungi | logs/.*/ansible/.* presumably, though yeah i can imagine for some builds that might be a massive number of log entries | 14:08 |
| fungi | but maybe no moreso than devstack's logs | 14:08 |
| mnasiadka | fungi: so - it's around 15 megabytes per job (https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_34c/695190/1/check/kolla-ansible-ubuntu-source/34cf558/primary/logs/ansible/) | 14:09 |
| mnasiadka | I don't know what is the usual devstack log size ;-) | 14:09 |
| mnasiadka | we could try although to fetch only the failed tasks into some other log file and push that to logstash | 14:10 |
| *** priteau has quit IRC | 14:11 | |
| fungi | or perhaps only archive them on failed builds | 14:12 |
| *** priteau has joined #openstack-infra | 14:12 | |
| *** pgaxatte has quit IRC | 14:12 | |
| mnasiadka | fungi: that's even better - is there some knob to turn? :) | 14:13 |
| fungi | yeah, it would be in your post playbook for the job defintion | 14:13 |
| *** priteau has quit IRC | 14:13 | |
| fungi | though be aware that tends to make for false positives if you match on omnipresent-but-benign log entries | 14:13 |
| fungi | since it's easy to think you've found the signature for all your failures because the entry only appears on build failures (because the log it's written to is only archived on failure) | 14:14 |
| *** jaosorior has quit IRC | 14:18 | |
| *** Lucas_Gray has quit IRC | 14:21 | |
| mnasiadka | fungi: got it | 14:21 |
| *** pgaxatte has joined #openstack-infra | 14:29 | |
| AJaeger | config-core, please review https://review.opendev.org/#/c/695124/ to help with the py3 grenade fun | 14:29 |
| *** PrinzElvis has quit IRC | 14:38 | |
| *** eharney has joined #openstack-infra | 14:39 | |
| *** goldyfruit_ has joined #openstack-infra | 14:43 | |
| *** kjackal has joined #openstack-infra | 14:45 | |
| *** jamesmcarthur has quit IRC | 14:49 | |
| dmsimard | clarkb, ianw: got latest version of ara working on bridge jobs: https://3753dea4ea75a3fe0692-37ad574b65e44b7a1b1058900a1af5e6.ssl.cf5.rackcdn.com/695108/7/check/system-config-run-backup/6e71391/bridge.openstack.org/ara-report/ | 14:56 |
| *** jklare has quit IRC | 14:57 | |
| *** jamesmcarthur has joined #openstack-infra | 14:57 | |
| *** rh-jelabarre has joined #openstack-infra | 14:58 | |
| *** dtantsur|brb is now known as dtantsur | 15:00 | |
| *** jklare has joined #openstack-infra | 15:01 | |
| *** dpawlik has quit IRC | 15:08 | |
| *** johnthetubaguy has joined #openstack-infra | 15:09 | |
| *** hrw has left #openstack-infra | 15:09 | |
| *** ociuhandu has joined #openstack-infra | 15:13 | |
| *** ociuhandu has quit IRC | 15:19 | |
| *** udesale has quit IRC | 15:20 | |
| *** udesale has joined #openstack-infra | 15:20 | |
| *** rlandy|rover is now known as rlandy|rover|mtg | 15:50 | |
| *** gfidente has quit IRC | 15:56 | |
| *** ociuhandu has joined #openstack-infra | 15:56 | |
| fungi | corvus: clarkb: anything in particular i should do with the afs volumes which have stale vldb locks other than remove the lock and redo the vos release with -localauth? they're locked for a release operation according to vos examine, with afs02.dfw unsurprisingly showing it's still at an old release | 15:59 |
| corvus | fungi: i think as long as the release isn't actually running, it's safe to unlock and redo. | 16:00 |
| *** kjackal has quit IRC | 16:00 | |
| fungi | yeah, these all got killed by timeout | 16:00 |
| *** rlandy|rover|mtg is now known as rlandy|rover | 16:01 | |
| *** ociuhandu has quit IRC | 16:01 | |
| clarkb | fungi: does that mean the local releasechange wasnt working? | 16:05 |
| *** armax has joined #openstack-infra | 16:05 | |
| fungi | clarkb: yeah, i think based on scrollback it needs some additional hacking to get puppet to stop overwriting the authorized_keys file ansible is setting | 16:06 |
| fungi | sounds like they're both fighting over it | 16:06 |
| fungi | i suppose we could create a separate shell user for that purpose | 16:07 |
| *** whoami-rajat_ has joined #openstack-infra | 16:08 | |
| *** iurygregory has quit IRC | 16:12 | |
| *** udesale has quit IRC | 16:14 | |
| *** gyee has joined #openstack-infra | 16:16 | |
| fungi | infra-root: i have a root screen session on mirror-update.openstack.org holding the update flock for the ubuntu mirror and have done `vos unlock mirror.ubuntu --localauth && vos release mirror.ubuntu --localauth` in a root screen session on afsdb01 | 16:17 |
| fungi | this will likely take a while as i'm expecting it to have to perform a full release to afs02.dfw | 16:18 |
| *** dpawlik has joined #openstack-infra | 16:18 | |
| gmann | AJaeger: thanks, still not sure how that happened (or zuul magic on branch variant ) so asked the question on ML for learning to me and other community members if such issue comes again. | 16:18 |
| fungi | once it's done, i'll start in on any other volumes in a similar situation | 16:18 |
| AJaeger | gmann: I just pointed corvus to it as well (on #zuul) | 16:19 |
| gmann | thanks | 16:22 |
| fungi | gonna disappear to grab lunch, then return and continue | 16:22 |
| *** ricolin has quit IRC | 16:24 | |
| AJaeger | gmann: want to ask stable team or barbican to review all changes? With the -1s from Zuul, you could ask an infra-root as well to force merge the changes to fix your problem. having stable team reviewed would be nice in that case | 16:29 |
| *** lmiccini has quit IRC | 16:29 | |
| gmann | AJaeger: yeah pinged barbican stable core and mriedem and tonyb also | 16:29 |
| *** dpawlik has quit IRC | 16:29 | |
| AJaeger | thanks, gmann | 16:30 |
| *** ociuhandu has joined #openstack-infra | 16:33 | |
| *** dpawlik has joined #openstack-infra | 16:35 | |
| *** ociuhandu has quit IRC | 16:38 | |
| *** ociuhandu has joined #openstack-infra | 16:38 | |
| *** jaosorior has joined #openstack-infra | 16:40 | |
| *** dpawlik has quit IRC | 16:41 | |
| *** pgaxatte has quit IRC | 16:51 | |
| *** goldyfruit___ has joined #openstack-infra | 16:53 | |
| *** goldyfruit___ is now known as goldyfruit | 16:54 | |
| *** goldyfruit_ has quit IRC | 16:56 | |
| *** gfidente has joined #openstack-infra | 16:57 | |
| *** lucasagomes has quit IRC | 16:57 | |
| *** dpawlik has joined #openstack-infra | 17:12 | |
| *** xek_ has joined #openstack-infra | 17:15 | |
| *** xek has quit IRC | 17:17 | |
| *** rpittau is now known as rpittau|afk | 17:18 | |
| *** dpawlik has quit IRC | 17:21 | |
| *** takamatsu has joined #openstack-infra | 17:23 | |
| *** ramishra has quit IRC | 17:23 | |
| *** michael-beaver has joined #openstack-infra | 17:25 | |
| *** goldyfruit has quit IRC | 17:27 | |
| *** gfidente is now known as gfidente|afk | 17:28 | |
| *** goldyfruit has joined #openstack-infra | 17:30 | |
| *** goldyfruit has quit IRC | 17:31 | |
| *** diablo_rojo has joined #openstack-infra | 17:35 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 17:35 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add build-container-image role https://review.opendev.org/695251 | 17:35 |
| *** dpawlik has joined #openstack-infra | 17:35 | |
| *** diablo_rojo has quit IRC | 17:40 | |
| clarkb | I've almost got inap's mirror set up with the new volume. Waiting on the old apache proxy dir to be cleaned out so that I can mount over the top of it cleanly | 17:47 |
| clarkb | I'll mount -a then restart apache and openafs-client and test then reboot and test. If happy at that point I think we can turn nodepool back on | 17:48 |
| fungi | sounds good, thanks! | 17:49 |
| fungi | the vos release for mirror.ubuntu is, surprising to no one i'm sure, still going | 17:49 |
| *** jaosorior has quit IRC | 17:50 | |
| *** dpawlik has quit IRC | 17:51 | |
| *** whoami-rajat_ is now known as whoami-rajat | 17:53 | |
| *** jaosorior has joined #openstack-infra | 17:57 | |
| *** igordc has joined #openstack-infra | 18:00 | |
| *** jpena is now known as jpena|off | 18:00 | |
| *** derekh has quit IRC | 18:01 | |
| clarkb | http://mirror.mtl01.inap.opendev.org/ubuntu/ seems to be working now after a reboot | 18:04 |
| clarkb | now to find the change to revert | 18:04 |
| fungi | clarkb: Change-Id: Iecc8e302f8d6ac374a81deaeddf14bcd997829b2 | 18:05 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Revert "Revert "Enable inap-mtl01"" https://review.opendev.org/695254 | 18:05 |
| clarkb | fungi: ^ | 18:09 |
| clarkb | fungi: do you think we should try doing concurrent vos releases? I wonder if each release would get its own udp window size or if it is shared. If shared one at a time is probably better. If independent then running them separately will likely speed stuff up | 18:11 |
| fungi | i suspect the bandwidth is being maxxed out by as many or as few as we run concurrently | 18:14 |
| fungi | but cacti would probably confirm | 18:14 |
| *** tesseract has quit IRC | 18:18 | |
| *** mattw4 has joined #openstack-infra | 18:19 | |
| *** jaosorior has quit IRC | 18:22 | |
| *** diablo_rojo has joined #openstack-infra | 18:23 | |
| clarkb | #status log Added 200GB cinder volume to mirror01.mtl01.inap.opendev.org to be used as openafs and apache2 cache backend. Filesystems created and mounted at the appropriate cache locations and services restarted. | 18:24 |
| openstackstatus | clarkb: finished logging | 18:24 |
| *** ralonsoh has quit IRC | 18:29 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "Enable inap-mtl01"" https://review.opendev.org/695254 | 18:30 |
| fungi | clarkb: yeah, i doubt we'll gain a lot by trying parallel runs: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=6405&rra_id=all | 18:37 |
| fungi | seems somewhat capped around 300mbps | 18:37 |
| *** diablo_rojo has quit IRC | 18:48 | |
| *** tosky has quit IRC | 18:55 | |
| clarkb | mnasiadka: fungi: things to keep in mind re indexing of logs. Those log files appear to be without timestamps. I've been pretty stringent about requiring those in the past if you want to index in elasticsearch. Without timestamps the timestamps yuo'll get out of logstash are meaningless (time of index rather than time of actual log occurance). We also do our best to index > DEBUG logs. This helps us | 18:55 |
| clarkb | avoid flooding the elasticsearch servers with records (we peak at over a billion a day) as well as ensures that the correct level of logging is exposed to users when errors occur | 18:55 |
| clarkb | mnasiadka: fungi I'm not sure how easy it will be to convince ansible to log sanely, btu at the very least we will want timestamps | 18:56 |
| mnasiadka | clarkb: noted - timestamps should not be problematic | 18:56 |
| clarkb | as a general note timestamps are also a requirement of third party CIs | 18:56 |
| clarkb | they make debugging jobs much easier | 18:56 |
| clarkb | not having them is a fairly major bug imo | 18:56 |
| *** ociuhandu has quit IRC | 18:57 | |
| *** ociuhandu has joined #openstack-infra | 18:59 | |
| clarkb | oh also because the format is different than one of the existing formats we parse (apache, syslog/journald, oslo, zuul console log) we may need to add a new parsing definition to the logstash rules | 19:01 |
| clarkb | however if you end up with logs that look like the zuul console log (which is basically timestamp + hostname + ansible output) then you can probably tell logstash it is a console log | 19:01 |
| *** ociuhandu has quit IRC | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:14 | |
| *** jamesmcarthur has joined #openstack-infra | 19:27 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 19:28 |
| *** lpetrut has quit IRC | 19:32 | |
| *** lpetrut has joined #openstack-infra | 19:32 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Explicit mention py2 for grenade py2 jobs https://review.opendev.org/695124 | 19:32 |
| clarkb | no certcheck warning for opendev today \o/ | 19:35 |
| *** ociuhandu has joined #openstack-infra | 19:35 | |
| AJaeger | great work, clarkb ! | 19:40 |
| *** pkopec has quit IRC | 19:41 | |
| fungi | excellent! | 19:44 |
| *** ociuhandu has quit IRC | 19:46 | |
| *** pkopec has joined #openstack-infra | 19:47 | |
| *** ociuhandu has joined #openstack-infra | 19:47 | |
| *** iurygregory has joined #openstack-infra | 19:48 | |
| *** ociuhandu has quit IRC | 19:53 | |
| *** sreejithp has joined #openstack-infra | 19:55 | |
| *** pkopec has quit IRC | 19:55 | |
| *** ociuhandu has joined #openstack-infra | 19:57 | |
| AJaeger | gmann: glad to see that the barbican job now works - but that might just be luck. We need to merge the change on all branches - so, https://review.opendev.org/689461 , https://review.opendev.org/689463 , https://review.opendev.org/689462 , https://review.opendev.org/689466 are needed as well. infra-root, should we force-merge those zuul updates to remoe a wrong branch matcher? | 19:58 |
| *** ociuhandu has quit IRC | 20:05 | |
| *** whoami-rajat has quit IRC | 20:08 | |
| *** eharney has quit IRC | 20:08 | |
| *** gfidente|afk has quit IRC | 20:12 | |
| *** jamesmcarthur has quit IRC | 20:20 | |
| ianw | fungi / clarkb : could you poke at https://review.opendev.org/#/c/695129/ to fix the half-deployed vos release key | 20:21 |
| fungi | absumotutely | 20:21 |
| *** jamesmcarthur has joined #openstack-infra | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| fungi | ianw: i'm probably unclear on the hostvars structure there, but see inline question | 20:26 |
| *** jamesmcarthur has joined #openstack-infra | 20:26 | |
| ianw | fungi: it should be; copy-paste error :/ | 20:27 |
| fungi | no worries | 20:28 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: vos-release: fix key sourcing; disable exclusive key https://review.opendev.org/695129 | 20:29 |
| tonyb | gmann: I must have missed that ping .... what do you need me to review? | 20:30 |
| tonyb | gmann: Oh I see it now in the nova channel, I'll do it today | 20:31 |
| *** ociuhandu has joined #openstack-infra | 20:36 | |
| *** iurygregory has quit IRC | 20:42 | |
| *** michael-beaver has quit IRC | 20:45 | |
| *** ociuhandu has quit IRC | 20:51 | |
| *** sreejithp has quit IRC | 20:52 | |
| *** ociuhandu has joined #openstack-infra | 20:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 20:54 |
| *** sreejithp has joined #openstack-infra | 20:57 | |
| *** ociuhandu has quit IRC | 21:06 | |
| *** eharney has joined #openstack-infra | 21:14 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 21:22 |
| *** xek__ has joined #openstack-infra | 21:30 | |
| *** xek_ has quit IRC | 21:32 | |
| *** dtantsur is now known as dtantsur|afk | 21:32 | |
| *** tosky has joined #openstack-infra | 21:41 | |
| *** pcaruana has quit IRC | 21:42 | |
| fungi | vos release of mirror.ubuntu finished. rerunning the cronjob for it now to make sure it completes smoothly | 21:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 21:47 |
| *** ociuhandu has joined #openstack-infra | 21:59 | |
| fungi | okay, http://files.openstack.org/mirror/ubuntu/timestamp.txt is caught up. parsing the vos listvldb output, i see 13 other volumes in a similar state (all mirrors) which also need this treatment | 22:04 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 22:04 |
| fungi | i'll knock out the rest of the ubuntu-related ones first | 22:05 |
| ianw | fungi: hrm 2 hours for a full release? | 22:05 |
| ianw | did you sprinkle magic dust? | 22:05 |
| fungi | it may not have been a full release after all | 22:06 |
| fungi | but it was sufficient for afs02.dfw to start showing current again | 22:07 |
| fungi | so i'll take it | 22:07 |
| *** ociuhandu has quit IRC | 22:07 | |
| ianw | huh, yeah that's the way it *should* work :) | 22:07 |
| fungi | anyway, mirror.ubuntu-ports is underway now | 22:07 |
| ianw | have you thought about moving reprepro to ansible? | 22:08 |
| ianw | how much magic is in the puppet? | 22:08 |
| fungi | we add a handful of conffiles in /etc and some cronjobs | 22:08 |
| fungi | so not much, i don't think | 22:09 |
| fungi | and install the package for reprepro | 22:09 |
| fungi | also a wrapper script | 22:09 |
| fungi | maybe log rotation? | 22:09 |
| fungi | but no, i haven't given thought to it yet | 22:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 22:18 |
| *** slaweq has quit IRC | 22:26 | |
| ianw | fungi: yeah, none of this vos-release via localauth stuff applies for mirror-update.openstack.org, that's what had me thinking about it | 22:28 |
| ianw | those tend to lock themselves as much as the opendev.org server | 22:28 |
| *** dave-mccowan has quit IRC | 22:29 | |
| *** rcernin has joined #openstack-infra | 22:31 | |
| *** xek_ has joined #openstack-infra | 22:34 | |
| *** slaweq has joined #openstack-infra | 22:35 | |
| *** xek__ has quit IRC | 22:36 | |
| *** slaweq has quit IRC | 22:40 | |
| *** kopecmartin is now known as kopecmartin|off | 22:42 | |
| *** sreejithp has quit IRC | 22:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: Vendor pytoml and remarshal https://review.opendev.org/695050 | 22:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add build-container-image role https://review.opendev.org/695251 | 22:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 22:48 |
| *** slaweq has joined #openstack-infra | 22:50 | |
| *** slaweq has quit IRC | 22:55 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| clarkb | ianw sorry I was dealing with tye last of the plumbing (I hope). eating lunch now then back to help | 23:01 |
| ianw | clarkb: oh, just the mention of the word plumbing sends a chill down my spine :) | 23:02 |
| ianw | when you think about it, your whole house is surrounded by pressurised containers just waiting to explode and cost you a small fortune :) | 23:02 |
| clarkb | As of fridqy I can confirm this fact | 23:03 |
| clarkb | our water main broke | 23:03 |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| openstackgerrit | Michael Johnson proposed openstack/diskimage-builder master: Add output for mis-configured element scripts https://review.opendev.org/680785 | 23:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: use-buildset-registry: Vendor pytoml and remarshal https://review.opendev.org/695050 | 23:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add build-container-image role https://review.opendev.org/695251 | 23:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: use-buildset-registry: add podman support https://review.opendev.org/695051 | 23:05 |
| *** slaweq has joined #openstack-infra | 23:06 | |
| ianw | clarkb: oh dear ... on your side or the company's side? | 23:06 |
| clarkb | my side | 23:06 |
| clarkb | more than a meter underground | 23:07 |
| clarkb | 1.3m or so | 23:07 |
| ianw | :( | 23:07 |
| ianw | i do know a good plumber; i don't think he works in your area though :) | 23:08 |
| *** xek_ has quit IRC | 23:08 | |
| *** slaweq has quit IRC | 23:10 | |
| clarkb | what we discovered is you have to do your own hole digging because you can get emergency weekend plumbers but not hole diggers | 23:11 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| clarkb | I just finished filling the hole today as we'll get freezing temps tonight | 23:11 |
| ianw | at least you can skip the gym? | 23:13 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:13 | |
| gmann | AJaeger: yeah, merging all is the target. | 23:14 |
| *** slaweq has quit IRC | 23:15 | |
| clarkb | ianw: ya I haven't been on the bike in like a week. Dirt throwing seems sufficient :) | 23:16 |
| clarkb | and now I'm back at the desktop. Anything I should be looking at? | 23:16 |
| clarkb | AJaeger: gmann re force merging if we get barbican to ack it first that would be ideal but I think we can do that | 23:16 |
| gmann | sure, let me check the status on patch. | 23:17 |
| ianw | clarkb: https://review.opendev.org/#/c/695129/ if you could; the role added yeterday is broken without it | 23:19 |
| clarkb | ianw: done | 23:20 |
| *** slaweq has joined #openstack-infra | 23:21 | |
| gmann | clarkb: AJaeger seems fedora_latest job failing consistently on barbican stable/stein. something wrong so let's wait for force merge. | 23:23 |
| *** slaweq has quit IRC | 23:26 | |
| clarkb | ianw: also TIL about with_inventory_hostname | 23:28 |
| ianw | clarkb: yeah, i learnt about it making the letsencrypt roles; each host that's renewing registers a var, then the "deploy dns records" walks across all the letsencrypt group hostnames to check for values | 23:30 |
| *** slaweq has joined #openstack-infra | 23:30 | |
| clarkb | gmann: https://review.opendev.org/#/admin/groups/1097,members looks like stable-maint-core and dave mccowan are who we need to bug? | 23:31 |
| clarkb | gmann: If we can't reach them in a reasonable amount of time we can still force push but it is always good to get the ack of thse responsible first | 23:31 |
| gmann | ok. | 23:33 |
| clarkb | gmann: are things broken for anyone but barbican? | 23:33 |
| clarkb | if not maybe devstack can proceed and that will be the carrot for barbican to review these changes? | 23:34 |
| gmann | clarkb: i have not checked yet. let me see check | 23:34 |
| *** slaweq has quit IRC | 23:35 | |
| gmann | other projects are good. https://review.opendev.org/#/q/status:open+branch:stable/stein | 23:35 |
| gmann | let me check log and ping dave mccowan | 23:36 |
| clarkb | sanity checking the inap mirror things continue to look good there | 23:36 |
| *** dchen has joined #openstack-infra | 23:36 | |
| *** slaweq has joined #openstack-infra | 23:37 | |
| openstackgerrit | Merged opendev/system-config master: vos-release: fix key sourcing; disable exclusive key https://review.opendev.org/695129 | 23:39 |
| gmann | seems it is broken since 8 months. i pinged on barbican channel- https://review.opendev.org/#/c/650415/ | 23:39 |
| *** slaweq has quit IRC | 23:45 | |
| *** kaisers1 has quit IRC | 23:48 | |
| *** slaweq has joined #openstack-infra | 23:48 | |
| *** kaisers has joined #openstack-infra | 23:48 | |
| *** slaweq has quit IRC | 23:53 | |
| *** mattw4 has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!