| guesswhat | seems that cinder does not work with interal,external, backend tls on ( selfsigned ), fails to SSLVerificationFailed to etcd ( cordinator.backend_url ) | 08:37 |
|---|---|---|
| guesswhat | any ideas? | 08:37 |
| jingvar | is copy CA enabled? | 08:59 |
| guesswhat | jingvar: yes, only cinder->etcd fails | 09:07 |
| guesswhat | i am using xena | 09:07 |
| jingvar | and what is inside cinder container? | 09:13 |
| jingvar | is there CA? is cinder configure properly? | 09:13 |
| guesswhat | a also tried to test curl --cacert /etc/ssl/certs/ca-certificates.crt https://<ipofetcd>:2739 | 09:34 |
| guesswhat | i am using defaults for cinder + this config https://pastebin.com/aWhyzHM2 | 09:36 |
| dtantsur | morning folks! | 10:26 |
| dtantsur | I have a bifrost change https://review.opendev.org/c/openstack/bifrost/+/820390/ that makes kolla quite upset | 10:26 |
| dtantsur | I'm not 100% sure how to proceed with that, nor can I find enough logs to debug it | 10:27 |
| dtantsur | I suspect the cause may be the assumption about which services bifrost starts | 10:27 |
| dtantsur | Help appreciated, otherwise we may break the bifrost job soon | 10:33 |
| mnasiadka | I think there's some problem in starting Ironic service - but since we are checking systemctl status of ironic-api and ironic-inspector instead of ironic service - it's hard to understand what happened ;-) | 10:34 |
| dtantsur | yeah, exactly | 10:34 |
| mnasiadka | dtantsur: https://github.com/openstack/kolla-ansible/blob/a338df77f1ea2c5c72b05c9525e03e98948f50d4/tests/get_logs.sh#L96 - you would need to change this and depends-on on the bifrost change | 10:34 |
| dtantsur | I also think it simply refused to start | 10:34 |
| dtantsur | okay, this sounds like an option | 10:34 |
| opendevreview | Dmitry Tantsur proposed openstack/kolla-ansible master: Prepare tests for the Ironic combined service https://review.opendev.org/c/openstack/kolla-ansible/+/820513 | 10:39 |
| dtantsur | okay, testing now | 10:41 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Finish removing Monasca Log Transformer https://review.opendev.org/c/openstack/kolla-ansible/+/820348 | 10:43 |
| kevko | morning \o/ | 11:01 |
| opendevreview | Merged openstack/kayobe master: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820211 | 11:04 |
| dtantsur | mnasiadka: this is not the most useful information I've ever received: https://1ff1337c58cfd2c7e047-bfdf0aa5980ea9561e0ca2e62d8db60b.ssl.cf1.rackcdn.com/820390/4/check/kolla-ansible-centos8s-source-bifrost/0e9d27b/primary/logs/kolla/ironic/systemd-status-ironic.txt | 11:22 |
| dtantsur | any ideas? | 11:22 |
| frickler | dtantsur: my next idea would be to check the journal, not sure if https://1ff1337c58cfd2c7e047-bfdf0aa5980ea9561e0ca2e62d8db60b.ssl.cf1.rackcdn.com/820390/4/check/kolla-ansible-centos8s-source-bifrost/0e9d27b/primary/logs/system_logs/journal/dbdb109d723142af9579c57cb5ac3a17/index.html is usable. for devstack I think we do some special stuff to export the journal and make it importable locally | 11:26 |
| dtantsur | frickler: you can read binary journals with 'journalctl --file', but the one I checked on the previous run didn't have any ironic stuff | 11:26 |
| guesswhat | jingvar, do you have any idea? thanks | 11:28 |
| yoctozepto | dtantsur: error 203 is for trying to execute non-executable things | 11:35 |
| dtantsur | huh | 11:36 |
| dtantsur | it doesn't make much sense to me | 11:36 |
| yoctozepto | I think it's when something is not `chmod +x`ed or the shebang uses a missing interpreter | 11:36 |
| yoctozepto | not sure if it's also 203 when the file itself is missing | 11:36 |
| dtantsur | yep, but these scripts are pbr-generated | 11:36 |
| yoctozepto | well, what can I say :-) | 11:37 |
| dtantsur | oh, well | 11:37 |
| yoctozepto | I just interpreted the error for you (-: | 11:37 |
| dtantsur | does this job respect Depends-On ironic patches | 11:37 |
| dtantsur | ? | 11:37 |
| yoctozepto | well, that's hard to say, for sure it does on bifrost; but perhaps bifrost part in kolla simply ignores it further | 11:38 |
| dtantsur | it's defined in the job though | 11:38 |
| dtantsur | and in your case may require explicit handling in kolla itself | 11:38 |
| yoctozepto | because my understanding is that bifrost tries to download ironic so I'm pretty sure our scripts don't care about that depends-on | 11:38 |
| yoctozepto | yeah | 11:39 |
| yoctozepto | so problem solved | 11:39 |
| yoctozepto | kind of | 11:39 |
| dtantsur | right. in bifrost we special-case zuul to make it work | 11:39 |
| dtantsur | yeah :) | 11:39 |
| dtantsur | we can check again once the ironic patch merges | 11:39 |
| yoctozepto | team work, dream work | 11:39 |
| dtantsur | true :) | 11:39 |
| dtantsur | thanks for checking with me | 11:39 |
| yoctozepto | yw :-) | 11:40 |
| guesswhat | guys, any idea why cinder fails to TLSVerification to etcd ( coordinator.backend_url ), anything else is working ( iam using internal,external, backend tls with import ca = selfsigned ) | 11:49 |
| dtantsur | yoctozepto: please do merge https://review.opendev.org/c/openstack/kolla-ansible/+/820513 though, it's a valid change nonetheless | 11:52 |
| jingvar | guesswhat: I've missed you messages, the chats history is small | 11:52 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/xena: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820499 | 12:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/wallaby: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820500 | 12:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/victoria: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820501 | 12:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/victoria: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820501 | 12:04 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/ussuri: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820502 | 12:05 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/train: Fix installation prefix detection https://review.opendev.org/c/openstack/kayobe/+/820503 | 12:06 |
| guesswhat | jingvar: curl --cacert /etc/ssl/certs/ca-certificates.crt https://<ipofetcd>:2739 fails to TLSverify too, I am using default cinder ( enable_cinder ) and this custom config https://pastebin.com/aWhyzHM2 ( everything else is working.. ) | 12:25 |
| guesswhat | *2379 | 12:25 |
| jingvar | same curl from another container? | 12:40 |
| guesswhat | jingvar: it does not work for even for other containers..., etcd is enabled, cuz of kuryr.. , but everyting else is working correctly no problems for other serverices calling API behind https | 13:25 |
| guesswhat | what is weird that its calling to node ip, not vip address | 13:26 |
| guesswhat | so probably some SAN for node ip of etcd is missing in certificates... | 13:27 |
| guesswhat | curl to https://172.25.244.100:8000/v1 and other APIs is working ,but curl to https://172.25.254.164:2379 is not, seems that etcd itself is running with https ( etcdserver: published {Name:openstack ClientURLs:[https://172.25.254.164:2379]} to cluster 1f4e3b4571ff585 ) | 13:30 |
| guesswhat | jingvar: these https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/etcd/defaults/main.yml#L21-L22 are standalone? kolla_copy_ca_into_containers does not affect these, right ? | 13:52 |
| yoctozepto | dtantsur: yes, thank you; and thanks for ensuring the gates of kolla remain green | 14:19 |
| opendevreview | Seena Fallah proposed openstack/kolla-ansible master: ovn: configure ovn in ovsdb only on ovn-controller hosts https://review.opendev.org/c/openstack/kolla-ansible/+/820544 | 15:49 |
| opendevreview | Seena Fallah proposed openstack/kolla-ansible master: ovn: configure ovn in ovsdb only on ovn-controller hosts https://review.opendev.org/c/openstack/kolla-ansible/+/820544 | 16:12 |
| opendevreview | Seena Fallah proposed openstack/kolla-ansible master: ovn: configure ovn in ovsdb only on ovn-controller hosts https://review.opendev.org/c/openstack/kolla-ansible/+/820544 | 16:13 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: Initialise Ansible collection https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820166 | 16:26 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: Import baremetal role from kolla-ansible https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820168 | 16:26 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: docs: remove most boilerplate, initialise contributor guide https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820560 | 16:26 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: docs: remove most boilerplate, initialise contributor guide https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820560 | 16:31 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: Import baremetal role from kolla-ansible https://review.opendev.org/c/openstack/ansible-collection-kolla/+/820168 | 16:31 |
| opendevreview | Merged openstack/kolla stable/xena: nova-compute: trim image a bit on CentOS https://review.opendev.org/c/openstack/kolla/+/820345 | 16:37 |
| opendevreview | Verification of a change to openstack/kolla stable/victoria failed: nova-compute: trim image a bit on CentOS https://review.opendev.org/c/openstack/kolla/+/820347 | 16:37 |
| guesswhat2 | guys, any idea why cinder fails to TLSVerification to etcd ( coordinator.backend_url ), anything else is working ( iam using internal,external, backend tls with import ca = selfsigned ) ] | 18:00 |
| jingvar | I don't see a code around etcd-cert.pem | 18:45 |
| jingvar | It looks like it doesnt work at all | 18:46 |
| jingvar | etcd-cert.pem was found in 2 files, kolla-ansible/ansible/roles/etcd/defaults/main.yml and kolla-ansible/ansible/roles/etcd/templates/etcd.json.j2 | 18:47 |
| jingvar | first one for ETCD_CERT_FILE and ETCD_PEER_CERT_FILE and them no more uses | 18:50 |
| jingvar | second one just containers config | 18:50 |
| guesswhat2 | jingvar: oh, so its broken, right ? | 18:50 |
| jingvar | I think yes | 18:51 |
| jingvar | cert file must goes from somewhere | 18:52 |
| jingvar | and must be delivered to appropriate services-containers | 18:52 |
| jingvar | I don't see something about this | 18:53 |
| jingvar | brave new world :) | 18:53 |
| jingvar | as I rigth understand etcds must have same certs | 18:56 |
| guesswhat2 | jingvar: why etcd is not proxied via haproxy ? | 18:56 |
| jingvar | etcd is database | 18:57 |
| jingvar | I think as mariadb galera clustuer it should have itself mechanism to sync | 18:57 |
| jingvar | etcd is internal function of kubernetes | 18:59 |
| jingvar | only kubeapi etc work with it | 18:59 |
| jingvar | no points to link with openstack | 19:00 |
| jingvar | I don't know freezer at all | 19:00 |
| guesswhat2 | jingvar: but etcd has rest api and client are interacting with it on layer7 | 19:00 |
| jingvar | for example? | 19:01 |
| guesswhat2 | its http grcp | 19:01 |
| jingvar | let me read about freezerr | 19:02 |
| guesswhat2 | why freezer? | 19:02 |
| jingvar | I dont know, | 19:05 |
| jingvar | which component you try to use | 19:06 |
| guesswhat2 | freezer does not work at all ( broken https://review.opendev.org/c/openstack/kolla-ansible/+/816363 ), this one is something different | 19:08 |
| guesswhat2 | everything that requires etcd does not work with TLS enabled, probably CA is not imported to containers | 19:08 |
| jingvar | etcd and cinder - appear from? | 19:09 |
| opendevreview | MargaritaShakhova proposed openstack/kolla-ansible master: Add ironic-inspector policy configuration https://review.opendev.org/c/openstack/kolla-ansible/+/820063 | 19:14 |
| jingvar | certs https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/etcd/tasks/copy-certs.yml#L19 | 19:15 |
| guesswhat2 | ERROR oslo_service.service requests.exceptions.SSLError: HTTPSConnectionPool(host='172.25.254.164', port=2379): Max retries exceeded with url: /v3alpha/lease/grant (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))) | 19:17 |
| guesswhat2 | jingvar ^ | 19:57 |
| opendevreview | Maksim Malchuk proposed openstack/kayobe master: Adds support for custom Placement configuration. https://review.opendev.org/c/openstack/kayobe/+/818755 | 20:52 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/train: Make broken Python 2 jobs non-voting https://review.opendev.org/c/openstack/kayobe/+/820601 | 21:57 |
| opendevreview | Merged openstack/kayobe master: Add dependencies for EFI and LVM based overcloud images https://review.opendev.org/c/openstack/kayobe/+/819887 | 23:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!