| *** hamalq has quit IRC | 00:07 | |
| *** auristor has quit IRC | 00:21 | |
| *** auristor has joined #opendev | 00:28 | |
| *** whoami-rajat has quit IRC | 01:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: remove extlinux/syslinux path https://review.opendev.org/c/openstack/diskimage-builder/+/541129 | 01:35 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Futher bootloader cleanups https://review.opendev.org/c/openstack/diskimage-builder/+/790878 | 01:35 |
| *** darshna has quit IRC | 01:51 | |
| *** tkajinam has quit IRC | 03:34 | |
| *** tkajinam has joined #opendev | 03:34 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Remove pbx.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/790895 | 03:46 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Remove pbx.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/790895 | 03:59 |
| *** ykarel has joined #opendev | 04:18 | |
| ianw | #status log Asterisk PBX service retired; see https://review.opendev.org/c/opendev/system-config/+/790190 | 04:39 |
| openstackstatus | ianw: finished logging | 04:39 |
| *** whoami-rajat_ has joined #opendev | 04:44 | |
| *** stevebaker has quit IRC | 04:52 | |
| *** hemanth_n has joined #opendev | 04:55 | |
| *** marios has joined #opendev | 05:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: bootloader: remove extlinux/syslinux path https://review.opendev.org/c/openstack/diskimage-builder/+/541129 | 05:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Futher bootloader cleanups https://review.opendev.org/c/openstack/diskimage-builder/+/790878 | 05:46 |
| openstackgerrit | Merged opendev/system-config master: Cleanup ssl_cert_check puppet components https://review.opendev.org/c/opendev/system-config/+/789652 | 06:02 |
| *** ralonsoh has joined #opendev | 06:07 | |
| *** slaweq has joined #opendev | 06:16 | |
| *** ralonsoh has quit IRC | 06:48 | |
| *** sboyron has joined #opendev | 06:48 | |
| *** ralonsoh has joined #opendev | 06:49 | |
| *** tinwood has quit IRC | 06:54 | |
| *** tinwood has joined #opendev | 06:57 | |
| *** fressi has joined #opendev | 07:00 | |
| *** ysandeep|away is now known as ysandeep | 07:05 | |
| *** jhesketh has quit IRC | 07:08 | |
| *** tosky has joined #opendev | 07:09 | |
| *** hashar has joined #opendev | 07:16 | |
| *** andrewbonney has joined #opendev | 07:17 | |
| *** rpittau|afk is now known as rpittau | 07:26 | |
| *** fressi has quit IRC | 07:38 | |
| *** fressi has joined #opendev | 07:44 | |
| *** amoralej|off is now known as amoralej | 07:46 | |
| *** jpena|off is now known as jpena | 07:59 | |
| openstackgerrit | Benjamin Schanzel proposed zuul/zuul-jobs master: Allow Specifying Log File Retention in s3 Uploader https://review.opendev.org/c/zuul/zuul-jobs/+/790050 | 08:06 |
| openstackgerrit | Merged openstack/diskimage-builder master: dib-lint: match text/x-script.python https://review.opendev.org/c/openstack/diskimage-builder/+/790369 | 08:09 |
| openstackgerrit | Merged openstack/diskimage-builder master: containerfile: automatically search for distro docker files https://review.opendev.org/c/openstack/diskimage-builder/+/790362 | 08:19 |
| openstackgerrit | Merged openstack/diskimage-builder master: bootloader: disable BLS for Fedora https://review.opendev.org/c/openstack/diskimage-builder/+/790549 | 08:19 |
| *** jhesketh has joined #opendev | 08:23 | |
| *** ysandeep is now known as ysandeep|lunch | 08:39 | |
| *** sshnaidm_ has joined #opendev | 08:48 | |
| *** sshnaidm has quit IRC | 08:50 | |
| openstackgerrit | Martin Kopec proposed opendev/irc-meetings master: Update interop-wg-meeting info https://review.opendev.org/c/opendev/irc-meetings/+/790923 | 08:54 |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| *** rpittau is now known as rpittau|bbl | 09:07 | |
| *** darshna has joined #opendev | 09:08 | |
| *** sshnaidm_ is now known as sshnaidm | 09:28 | |
| openstackgerrit | Dmitriy Rabotyagov proposed opendev/lodgeit master: Add py3.8 support https://review.opendev.org/c/opendev/lodgeit/+/773479 | 10:00 |
| openstackgerrit | Dmitriy Rabotyagov proposed opendev/lodgeit master: Redesign manage.py to not use deprecated werkzeug.script https://review.opendev.org/c/opendev/lodgeit/+/693378 | 10:09 |
| *** rpittau|bbl is now known as rpittau | 10:19 | |
| *** hashar has quit IRC | 10:30 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:46 | |
| *** hashar has joined #opendev | 11:08 | |
| *** dtantsur is now known as dtantsur|brb | 11:19 | |
| *** ysandeep|lunch is now known as ysandeep | 11:19 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| frickler | infra-root: we seem to have a significant number of jobs with post_failure where no logs have been uploaded. sadly no time to dig myself today | 12:17 |
| frickler | https://zuul.opendev.org/t/openstack/build/65247750404549a0be5c858716045c33 is one example, but some more can be found e.g. on the status page currently in gate failures | 12:18 |
| *** sshnaidm|afk is now known as sshnaidm | 12:19 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| fungi | thanks, probably one of the swift providers erroring on auth or write. will try to find it | 12:29 |
| fungi | looks like your example ran from ze10 | 12:32 |
| fungi | WARNING:keystoneauth.identity.generic.base:Failed to discover available identity versions when contacting https://auth.cloud.ovh.net/ | 12:34 |
| fungi | nothing obvious in progress at http://travaux.ovh.net/ | 12:37 |
| fungi | yeah, looks like it's still happening based on a survey of timestamps in our executor lgs | 12:41 |
| fungi | TimeoutError is what's being raised | 12:41 |
| fungi | i'll get an emergency patch in to stop uploading logs to ovh | 12:42 |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Temporarily stop uploading logs to OVH https://review.opendev.org/c/opendev/base-jobs/+/790961 | 12:48 |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Revert "Temporarily stop uploading logs to OVH" https://review.opendev.org/c/opendev/base-jobs/+/790962 | 12:48 |
| fungi | infra-root: i'm going to bypass gating on the first of those and wip the second for now | 12:48 |
| *** amoralej is now known as amoralej|lunch | 12:55 | |
| openstackgerrit | Merged opendev/base-jobs master: Temporarily stop uploading logs to OVH https://review.opendev.org/c/opendev/base-jobs/+/790961 | 12:55 |
| fungi | #status log Bypassed gating to merge https://review.opendev.org/790961 for temporarily disabling log uploads to one of our providers | 12:56 |
| *** ykarel_ has joined #opendev | 12:56 | |
| openstackstatus | fungi: finished logging | 12:56 |
| *** ykarel has quit IRC | 12:58 | |
| *** ykarel_ is now known as ykarel | 13:00 | |
| *** rosmaita has joined #opendev | 13:05 | |
| *** whoami-rajat_ is now known as whoami-rajat | 13:17 | |
| *** timburke has quit IRC | 13:26 | |
| *** timburke has joined #opendev | 13:27 | |
| *** hemanth_n has quit IRC | 13:28 | |
| *** amoralej|lunch is now known as amoralej | 13:36 | |
| *** avass has quit IRC | 13:39 | |
| *** avass has joined #opendev | 13:39 | |
| *** calcmandan has quit IRC | 13:48 | |
| *** calcmandan has joined #opendev | 13:48 | |
| fungi | headed out to run some errands, should be back in an hour and will take a closer look at the ovh swift situation assuming nothing else catches fire in the meantime | 14:04 |
| *** fressi has quit IRC | 14:04 | |
| *** hashar is now known as hasharAway | 14:09 | |
| *** ykarel_ has joined #opendev | 14:14 | |
| *** ykarel has quit IRC | 14:16 | |
| *** ykarel_ is now known as ykarel | 14:17 | |
| *** lucasagomes has joined #opendev | 14:20 | |
| *** dtantsur|brb is now known as dtantsur | 14:26 | |
| *** rosmaita has left #opendev | 14:53 | |
| *** pongboom2 has quit IRC | 14:58 | |
| fungi | okay, back and catching up again | 15:10 |
| *** hasharAway is now known as hashar | 15:18 | |
| *** mlavalle has joined #opendev | 15:22 | |
| fungi | looks like all the builds which selected ovh for log uploads have finally completed as of ~50 minutes ago | 15:29 |
| fungi | #status notice Any builds with POST_FAILURE result and no available logs between 11:41 and 14:41 UTC today were related to an authentication endpoint problem in one of our providers and can be safely rechecked now | 15:30 |
| openstackstatus | fungi: sending notice | 15:30 |
| -openstackstatus- NOTICE: Any builds with POST_FAILURE result and no available logs between 11:41 and 14:41 UTC today were related to an authentication endpoint problem in one of our providers and can be safely rechecked now | 15:30 | |
| fungi | it looks like there was also a much shorter incident around 08:00 but i only counted a few builds which hit it | 15:31 |
| fungi | looks like the latest outage could be related to http://travaux.ovh.net/?do=details&id=50472 which is now marked "closed" but unfortunately the timing is unclear and the reason given is just "done" | 15:33 |
| openstackstatus | fungi: finished sending notice | 15:33 |
| *** ysandeep is now known as ysandeep|dinner | 15:34 | |
| fungi | i'll exercise base-test a bit and see if i get a successful upload to ovh, then we can approve the revert | 15:34 |
| *** ykarel has quit IRC | 15:37 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: DNM: exercise base-test https://review.opendev.org/c/opendev/base-jobs/+/791014 | 15:39 |
| Tengu | &16 | 15:40 |
| *** ysandeep|dinner is now known as ysandeep | 15:45 | |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add zuul02 https://review.opendev.org/c/opendev/zone-opendev.org/+/790480 | 16:03 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Swap zuul.opendev.org CNAME to zuul02.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/790482 | 16:03 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Reset zuul.o.o CNAME TTL to default https://review.opendev.org/c/opendev/zone-opendev.org/+/790483 | 16:03 |
| *** lucasagomes has quit IRC | 16:03 | |
| clarkb | fungi: do you think we can approve https://review.opendev.org/c/opendev/zone-opendev.org/+/790480/2 to avoid more merge conflicts? | 16:04 |
| clarkb | Then I'll plan to land https://review.opendev.org/c/opendev/system-config/+/790481 tomorrow first thing assuming my head is functionaing properly | 16:04 |
| clarkb | (and maybe we'll finish that swap tomorrow) | 16:04 |
| *** marios is now known as marios|out | 16:05 | |
| fungi | done | 16:07 |
| clarkb | thanks! | 16:07 |
| clarkb | and if you haven't yet looking over https://etherpad.opendev.org/p/opendev-zuul-server-swap today would be great so we can call out any flaws with that plan | 16:08 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add zuul02 https://review.opendev.org/c/opendev/zone-opendev.org/+/790480 | 16:09 |
| *** slaweq has quit IRC | 16:11 | |
| fungi | absolutely! | 16:12 |
| *** rpittau is now known as rpittau|afk | 16:12 | |
| *** marios|out has quit IRC | 16:24 | |
| *** ralonsoh has quit IRC | 17:02 | |
| *** jpena is now known as jpena|off | 17:07 | |
| *** hashar has quit IRC | 17:10 | |
| *** hashar has joined #opendev | 17:11 | |
| *** ysandeep is now known as ysandeep|away | 17:27 | |
| *** amoralej is now known as amoralej|off | 17:34 | |
| fungi | clarkb: not sure if you saw ianw's suggestion on 790481, but adding the new zuul server's inventory fqdn to the le altnames, if for no other reason than consistency. in a followup though, happy to push one if you haven't already | 17:48 |
| *** andrewbonney has quit IRC | 17:56 | |
| *** dtantsur is now known as dtantsur|afk | 18:01 | |
| *** avass has quit IRC | 18:03 | |
| *** hashar has quit IRC | 18:04 | |
| *** avass has joined #opendev | 18:09 | |
| melwitt | huh, I just noticed going to www.opendev.org fails "www.opendev.org’s DNS address could not be found" | 18:12 |
| openstackgerrit | Ade Lee proposed zuul/zuul-jobs master: Add role to enable FIPS on a node https://review.opendev.org/c/zuul/zuul-jobs/+/788778 | 18:17 |
| fungi | melwitt: you're right, there currently isn't one, but if we add one in https://opendev.org/opendev/zone-opendev.org/src/branch/master/zones/opendev.org/zone.db as a cname we'd want to make sure to add a redirect in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea/templates/gitea.vhost.j2 and altname for the certs in | 18:19 |
| fungi | https://opendev.org/opendev/system-config/src/branch/master/inventory/service/host_vars/gitea01.opendev.org.yaml through 08 | 18:19 |
| melwitt | ohh ok | 18:20 |
| fungi | thankfully that would just be two changes in gerrit (the system-config change, and then the zone-opendev.org change setting depends-on the other) | 18:20 |
| melwitt | ok cool | 18:21 |
| openstackgerrit | Martin Kopec proposed opendev/irc-meetings master: Update interop-wg-meeting info https://review.opendev.org/c/opendev/irc-meetings/+/790923 | 18:23 |
| fungi | melwitt: did you follow a link from somewhere to the www name, or just tried it on a whim? | 18:26 |
| melwitt | fungi: tried it while testing something, needed dummy url so I just put www.opendev.org | 18:27 |
| fungi | ahh, okay | 18:28 |
| fungi | frankly, we probably would have added it originally except we were doing direct layer-4 forwarding to gitea which didn't have an internal redirect mechanism, so would needed to have tacked on a separate system to handle the layer-7 redirects | 18:29 |
| melwitt | oh ok | 18:30 |
| fungi | since then we injected apache in between haproxy and gitea to do ua-based filtering so we could deal with some misbehaving webcrawlers | 18:30 |
| fungi | so if we're okay with the idea of apache being there permanently, or at least something like it, then extending it to do redirects is likely fine | 18:31 |
| fungi | alternatively we could host the redirect on a separate server, maybe static.opendev.org | 18:31 |
| clarkb | fungi: we should probably do that update to the actual change | 18:39 |
| clarkb | fungi: we also need a dns update for that altname | 18:39 |
| clarkb | the reason I'm thinking we shouldn't do a followup is we'll generate one too many extra certs that we don't need | 18:40 |
| fungi | clarkb: oh, yep for the acme cname | 18:40 |
| fungi | system-config-run-base failed in the gate on it anyway | 18:40 |
| clarkb | oh I see you approved the change, but it failed | 18:40 |
| clarkb | ya | 18:40 |
| clarkb | I can update it and push a zone update too | 18:40 |
| fungi | wfm, at the ready to reapprove | 18:41 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Swap zuul.opendev.org CNAME to zuul02.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/790482 | 18:45 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Reset zuul.o.o CNAME TTL to default https://review.opendev.org/c/opendev/zone-opendev.org/+/790483 | 18:45 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Add zuul02 acme cname https://review.opendev.org/c/opendev/zone-opendev.org/+/791039 | 18:45 |
| clarkb | fungi: https://review.opendev.org/c/opendev/zone-opendev.org/+/791039 that is the dns update | 18:45 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add zuul02 to inventory https://review.opendev.org/c/opendev/system-config/+/790481 | 18:47 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Clean up zuul01 from inventory https://review.opendev.org/c/opendev/system-config/+/790484 | 18:47 |
| clarkb | fungi: ^ and that should update the le stuff if I did it correctly | 18:47 |
| clarkb | nope I think I did something wrong | 18:48 |
| clarkb | oh wait no its fine I was looking at the child change | 18:48 |
| clarkb | double check me (also my brain isn't working so great) | 18:49 |
| fungi | yep, that looks correct to me | 18:50 |
| corvus | clarkb: qq why do we want a zuul02 cert? | 18:51 |
| corvus | don't we have the le stuff set up so we can give zuul02 the zuul.o.o cert? | 18:51 |
| fungi | corvus: it's more for consistency so we can directly address the inventory fqdn of servers for testing connectivity when we have more than one | 18:52 |
| fungi | just ends up as a subject altname on the final cert | 18:53 |
| corvus | fungi: ok, we don't do that for zuul01 but i guess it's no time like the present to start. | 18:53 |
| fungi | corvus: yep, also zuul01 was in a weird state living in the openstack.org domain officially, so we didn't add records to that zone which weren't absolutely necessary, for a variety of reasons | 18:54 |
| fungi | for stuff in opendev.org it's trivial to include the inventory fqdns on certs | 18:54 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add zuul02 acme cname https://review.opendev.org/c/opendev/zone-opendev.org/+/791039 | 18:56 |
| clarkb | ya I think its mostly for consistency | 18:56 |
| fungi | right, though it might come in more handy after 5.0 | 18:57 |
| fungi | when we can expect to have more than one scheduler/web server | 18:57 |
| clarkb | ++ | 18:57 |
| fungi | outside of upgrade replacement transition periods i mean | 18:57 |
| fungi | doing that for the gitea certs has proven itself very useful, since it gives us an easy way for users to identify which backend they're hitting through a load balancer | 18:58 |
| corvus | i find it interesting we've never felt the need to scale out zuul-web | 18:59 |
| fungi | i find it reassuring we've never needed to scale out zuul-web, that's for sure | 19:00 |
| fungi | seems to be highly efficient | 19:00 |
| openstackgerrit | Martin Kopec proposed opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/791043 | 19:04 |
| *** stevebaker has joined #opendev | 19:28 | |
| openstackgerrit | Jeremy Stanley proposed opendev/bindep master: DNM: Exercise base-test job https://review.opendev.org/c/opendev/bindep/+/791048 | 19:36 |
| *** hashar has joined #opendev | 19:43 | |
| *** slaweq has joined #opendev | 19:48 | |
| *** Jeffrey4l has quit IRC | 19:49 | |
| *** Jeffrey4l has joined #opendev | 20:00 | |
| *** sboyron has quit IRC | 20:13 | |
| spotz | fungi clarkb - can you leet my RDO announcement through on openstack-discuss:) | 20:18 |
| clarkb | spotz: fungi is a list admin for that one and should be able to | 20:29 |
| fungi | spotz: i did roughly an hour ago | 20:29 |
| spotz | Ha sweet! | 20:29 |
| spotz | It's so hhard cause you don't see your own posts | 20:30 |
| fungi | i think that's a subscription setting, by default you receive copies of your own posts | 20:33 |
| fungi | but you can set your subscription to exclude them | 20:33 |
| clarkb | I have rechecked https://review.opendev.org/c/opendev/system-config/+/790481 looks like apache on the scheduler failed to start (I would expect and LE problem to have bubbled up sooner in the playbook?) I don't know I have much ability to debug further today, but maybe someone else can take a look if it fails again | 20:39 |
| fungi | i've finished cooking/eating so can try to keep an eye on it | 20:43 |
| fungi | https://review.opendev.org/791048 has 5 builds which successfully uploaded to ovh swift, so i've approved 790962 now | 20:47 |
| openstackgerrit | Merged opendev/base-jobs master: Revert "Temporarily stop uploading logs to OVH" https://review.opendev.org/c/opendev/base-jobs/+/790962 | 20:52 |
| openstackgerrit | Merged opendev/irc-meetings master: Update interop-wg-meeting info https://review.opendev.org/c/opendev/irc-meetings/+/790923 | 21:02 |
| *** slaweq has quit IRC | 21:04 | |
| *** hashar has quit IRC | 21:13 | |
| openstackgerrit | Merged openstack/project-config master: Retire puppet-glare - Step 3: Remove Project https://review.opendev.org/c/openstack/project-config/+/790083 | 21:29 |
| fungi | clarkb: same job failed on the rerun... looking now to see if it's the same behavior | 21:32 |
| fungi | yup | 21:35 |
| fungi | Unable to reload service apache2: Job for apache2.service failed. | 21:35 |
| *** openstackgerrit has quit IRC | 21:47 | |
| ianw | sometimes we can see that if letsencrypt failed, there are some failure returns i think we don't catch correctly | 21:49 |
| ianw | the acme log should reveal ... looking | 21:49 |
| ianw | it seems we didn't collect it, nor apache logs | 21:51 |
| *** openstackgerrit has joined #opendev | 21:58 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: zuul job : collect some more logs https://review.opendev.org/c/opendev/system-config/+/791055 | 21:58 |
| clarkb | I wonder if I got something wrong with the zuul02.opendev.org altname addition? | 21:58 |
| fungi | ahh, thanks, i got sidetracked with other stuff before i could dig deeper | 21:58 |
| clarkb | though looking at it I don't see anything obviously wrong | 22:00 |
| clarkb | I guess we wait for the logs | 22:00 |
| *** openstackgerrit has quit IRC | 22:04 | |
| *** timburke has quit IRC | 22:08 | |
| *** timburke has joined #opendev | 22:09 | |
| ianw | Creating certs in /etc/letsencrypt-certs/zuul02.opendev.org | 22:23 |
| ianw | that bit looks right | 22:23 |
| *** darshna has quit IRC | 22:30 | |
| ianw | SSLCertificateFile: file '/etc/letsencrypt-certs/zuul.opendev.org/zuul.opendev.org.cer' does not exist or is empty | 22:38 |
| ianw | should be zuul02.opendev.org | 22:38 |
| *** openstackgerrit has joined #opendev | 22:39 | |
| openstackgerrit | Merged opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/791043 | 22:39 |
| ianw | i have to do school run but can fix up ./zuul-web/templates/openstack.vhost.j2 after if you like | 22:41 |
| *** tosky has quit IRC | 23:01 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: zuul-web : use hostname for LE cert https://review.opendev.org/c/opendev/system-config/+/791060 | 23:28 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: zuul-web : use hostname for LE cert https://review.opendev.org/c/opendev/system-config/+/791060 | 23:32 |
| fungi | aha, cool i was just looking at what needed to be edited there | 23:34 |
| ianw | i think if that's ok, remerge ontop of it and it should "just work" with the new hostname | 23:36 |
| fungi | yep | 23:37 |
| fungi | seems like it should be | 23:37 |
| *** openstackgerrit has quit IRC | 23:49 | |
| *** mlavalle has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!