| opendevreview | Merged openstack/openstack-ansible-nspawn_hosts stable/stein: Remove Debian Stable jobs https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/804309 | 03:00 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-nspawn_hosts stable/train: Remove Debian Stable jobs https://review.opendev.org/c/openstack/openstack-ansible-nspawn_hosts/+/804308 | 03:00 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 03:27 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_trove stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/814338 | 03:31 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_blazar stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/814339 | 03:32 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_congress stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_congress/+/814340 | 03:34 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_designate stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/814341 | 03:35 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_heat stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/814342 | 03:37 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_ironic stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/814343 | 03:39 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_magnum stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/814344 | 03:40 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_masakari stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/814345 | 03:41 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_mistral stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/814346 | 03:42 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_octavia stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/814347 | 03:44 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_rally stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/814348 | 03:47 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_sahara stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/814349 | 03:49 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_swift stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/814350 | 03:51 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_tacker stable/stein: The "stable" this originally referred to is stretch, which is now two "stable"'s ago. We've since realised that giving nodesets generic names doesn't work so well for this reason and will use codenames in the future (debian-bullseye, etc.). https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/814351 | 03:52 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_barbican stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/814337 | 04:13 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_blazar stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/814339 | 04:13 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_congress stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_congress/+/814340 | 04:13 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_designate stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/814341 | 04:14 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_heat stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/814342 | 04:14 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_ironic stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/814343 | 04:14 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_magnum stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/814344 | 04:14 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_masakari stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/814345 | 04:15 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_mistral stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/814346 | 04:15 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_octavia stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/814347 | 04:16 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_rally stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/814348 | 04:16 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_sahara stable/stein: Remove debian-stable jobs https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/814349 | 04:16 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_swift stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/814350 | 04:17 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_tacker stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/814351 | 04:17 |
| opendevreview | Ian Wienand proposed openstack/openstack-ansible-os_trove stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/814338 | 04:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest stable/victoria: Pin neutron-tempest-plugin to v1.6.0 https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814258 | 07:48 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-haproxy_server master: Add option to use alernative CA server for certbot https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/814364 | 07:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/stein: Update ca-certificates package before bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/814371 | 08:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari stable/stein: Remove debian-stable job https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/814345 | 08:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/stein: Update requests package before bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/814371 | 09:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/stein: Update requests package before bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/814371 | 09:13 |
| opendevreview | Martin Kopec proposed openstack/openstack-ansible-os_tempest master: python-tempestconf moved from osf/ to openinfra https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/814404 | 11:02 |
| *** akahat is now known as akahat|afk | 13:57 | |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_octavia master: Do not log private key https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/814430 | 14:17 |
| *** akahat|afk is now known as akahat | 14:48 | |
| *** frenzy_friday is now known as frenzyfriday|pto | 15:55 | |
| spatel | https://opendev.org/openstack/openstack-ansible-os_neutron/': The certificate issuer's certificate has expired | 16:22 |
| spatel | very odd | 16:23 |
| spatel | cert expire date is December 2021 | 16:24 |
| spatel | jrosser any idea? | 16:27 |
| jrosser | you mean thats what your browser says? | 16:28 |
| spatel | this command complaining - git clone https://opendev.org/openstack/openstack-ansible-os_neutron openstack-ansible-os_neutron-dpdk | 16:30 |
| spatel | trying to make a patch and got cert expire error during git clone | 16:30 |
| jrosser | i would guess that whatever system you are using needs its trust store updated to account for the expiry of the old LE root CA | 16:30 |
| jrosser | https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ | 16:31 |
| jrosser | or openssl needs updating, as they say there | 16:32 |
| spatel | hmm | 16:32 |
| ianw | speaking of cert errors ... | 21:11 |
| ianw | noonedeadpunk: there's a lot of -1 in https://review.opendev.org/q/topic:%22debian-stable-rm%22+(status:open%20OR%20status:merged) ... | 21:11 |
| ianw | it looks like centos7 is having issues | 21:11 |
| ianw | /opt/ansible-runtime/bin/pip install --isolated --index-url ... blah balh | 21:12 |
| ianw | SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) | 21:12 |
| ianw | i wonder if this is related to our images or something else | 21:12 |
| ianw | fungi: ^ any ideas? | 21:12 |
| ianw | 2021-10-16 11:59:50.904 | ca-certificates noarch 2021.2.50-72.el7_9 updates 379 k | 21:22 |
| ianw | according to what i read, this package has the right certificates and that's what we're installing in the image | 21:22 |
| fungi | ianw: sorry, you caught me in the midst of a meal, but is it possible our centos-7 images are very outdated? | 21:40 |
| fungi | though i guess you're saying 2021.2.50-72.el7_9 is recent enough | 21:40 |
| ianw | fungi: np, please don't skip meals for centos-7 issues :) | 21:40 |
| ianw | yeah, that looks right, and all images look up-to-date | 21:41 |
| ianw | interestingly this has | 21:50 |
| ianw | /opt/ansible-runtime/bin/pip install --isolated --index-url http://mirror.gra1.ovh.opendev.org/pypi/simple --trusted-host mirror.gra1.ovh.opendev.org --extra-index-url http://mirror.gra1.ovh.opendev.org/wheel/centos-7-x86_64 | 21:50 |
| ianw | but we redirect http->https so it upgrades | 21:50 |
| ianw | the url causing this is !!!!!! https://opendev.org/openstack/requirements/raw/stable/stein/upper-constraints.txt | 21:50 |
| ianw | (!!!) is my output, sorry | 21:50 |
| ianw | curl has no problem getting this | 21:51 |
| ianw | somehow requests in a virtualenv must bundle it's own certs or something? | 21:51 |
| ianw | ... that would be yes -> /opt/ansible-runtime/lib/python2.7/site-packages/pip/_vendor/requests/cacert.pem | 21:53 |
| jrosser | via certifi I think? | 21:53 |
| ianw | so pip vendors a cacert.pem and now can't update itself | 21:54 |
| jrosser | requests never uses the system trust store and it’s a huge pain | 21:55 |
| jrosser | REQUESTS_CA_BUNDLE env var can point it elsewhere though | 21:56 |
| ianw | hrm: "/opt/ansible-runtime/bin/pip install --upgrade pip" works | 21:56 |
| ianw | it's something to do with the "--constraints" flag | 21:59 |
| ianw | /opt/ansible-runtime/bin/pip install --isolated --index-url http://mirror.gra1.ovh.opendev.org/pypi/simple --trusted-host mirror.gra1.ovh.opendev.org --extra-index-url http://mirror.gra1.ovh.opendev.org/wheel/centos-7-x86_64 --upgrade pip setuptools wheel | 21:59 |
| ianw | works | 22:00 |
| ianw | although ... the constraints flag is the one that we redirect to the https site | 22:00 |
| jrosser | is it failing to understand the new LE root CA for getting upper-constraints? | 22:00 |
| fungi | sorry, back now, and yes that's what i was wondering | 22:01 |
| ianw | sigh, so it seems pip bundles cacert.pem | 22:01 |
| fungi | got it, so the problem is pinning to an old version of pip | 22:02 |
| ianw | but, this is python2 for centos7, and so even the latest pip is 20.3.4 which *still* doesn't have the right cert | 22:02 |
| fungi | time only moves forward, never backward. sorry to be a bummer :/ | 22:02 |
| fungi | oh, python 2.7? yeah there was some point release which added a cert bundle to the interpreter itself as a backport from 3.x | 22:03 |
| jrosser | it would be interesting to know if pointing requests at the system trust store is sufficient to fix it | 22:06 |
| fungi | i'm guessing it would be, yeah | 22:08 |
| jrosser | super late here, let me know if this needs further work and I can look tomorrow | 22:08 |
| fungi | i'm trying to dig up the relevant release note for whichever 2.7.x i was remembering | 22:09 |
| ianw | the other option is to add --trusted-hosts=opendev.org | 22:10 |
| ianw | for some reason it already does this for mirrors @ https://opendev.org/openstack/openstack-ansible/blame/branch/stable/stein/scripts/scripts-library.sh#L99 | 22:10 |
| fungi | i think it's a holdover from before we added https for them | 22:11 |
| fungi | at one point pip decided it didn't like http mirrors | 22:12 |
| fungi | mmm, pep 476 seems to be a breadcrumb, and later pep 493 | 22:14 |
| ianw | this only happens in a virtualenv on centos7 with python2 | 22:14 |
| ianw | things sort of work i guess because pypi is still valid | 22:15 |
| ianw | that looks like a digicert | 22:15 |
| ianw | so it's narrowed to python2+virtualenv+talking to LE sites | 22:16 |
| fungi | "PEP 476 updated httplib and modules which use it, such as urllib2 and xmlrpclib, to now verify that the server presents a certificate which is signed by a Certificate Authority in the platform trust store and whose hostname matches the hostname being requested by default, significantly improving security for many applications. This change was made in the Python 2.7.9 release." | 22:16 |
| fungi | so 2.7.9 was the transition i was remembering | 22:17 |
| fungi | this is probably relevant, since it's about rhel 7 and that specific transition: https://access.redhat.com/articles/2039753 | 22:17 |
| fungi | "The ssl module now checks the PYTHONHTTPSVERIFY environment variable - if set, its value overrides the settings from cert-verification.cfg. The value of 0 disables certificate verification and any other value enables it. This feature can be used by end users to enable or disable verification for a specific Python program, or a specific invocation of a Python program, without needing | 22:18 |
| fungi | to modify the program's source code." | 22:18 |
| fungi | so maybe that represents a possible workaround on centos-7's python 2.7 implementation as well | 22:19 |
| fungi | "By default, the Python ssl module uses the system CA certificate bundle - /etc/pki/tls/certs/ca-bundle.crt - shipped as part of the ca-certificates package. Inside corporate intranets, servers commonly use certificates issued by an internal corporate CA rather than by a public Internet CA. Any affected programs should be configured to use the internal CA certificate to be able to | 22:21 |
| fungi | successfully verify certificates of such servers." | 22:21 |
| ianw | fungi: yeah, the problem is that in a virtualenv pip is using it's bundled requests | 22:22 |
| ianw | it looks like | 22:22 |
| ianw | [global] | 22:22 |
| ianw | cert = /etc/pki/tls/certs/ca-bundle.crt | 22:22 |
| ianw | seems to make it work | 22:22 |
| fungi | right, and the bundled version of requests vendors in its on trust store? | 22:23 |
| ianw | yep | 22:23 |
| fungi | yeah, that seems like a straightforward solution | 22:23 |
| ianw | which i bet was put in to work around some crappy platform that didn't have the right certs for pip to talk to pypi or something | 22:24 |
| ianw | and now pip is the crappy platform that can't talk to letsencrypt | 22:24 |
| ianw | swings and roundabouts | 22:24 |
| fungi | and snakes and ladders | 22:24 |
| ianw | hrm, we're already installing a pip.conf ... | 22:27 |
| ianw | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/templates/etc/pip.conf.j2 | 22:27 |
| ianw | https://bugzilla.redhat.com/show_bug.cgi?id=2015326 | 23:29 |
| ianw | oh i think i see the issue here. it's actually openssl 1.0.2 | 23:56 |
| ianw | https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/ | 23:57 |
| ianw | it's not really the .pem file. it's openssl on centos7 not ignoring the expired certificate | 23:57 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!