| opendevreview | Bjoern Teipel proposed openstack/openstack-ansible-os_octavia master: Update amphora image creation https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/810136 | 01:06 |
|---|---|---|
| *** rpittau|afk is now known as rpittau | 07:24 | |
| *** frenzy_friday is now known as anbanerj|ruck | 10:36 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Clean up debian blazar_distro_packages https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/810183 | 11:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Clean up debian blazar_distro_packages https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/810183 | 11:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_blazar master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/809746 | 11:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/810184 | 11:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/810185 | 11:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/810187 | 11:33 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_heat master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/810188 | 11:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/810210 | 11:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/810219 | 12:39 |
| spatel | noonedeadpunk around | 12:39 |
| spatel | who generate this certs? - /etc/openstack_deploy/pki/certs/certs/ | 12:40 |
| noonedeadpunk | pki role? | 12:40 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/certificate-authority.yml | 12:41 |
| spatel | I have deleted all certs in that directory and re-run playbook which did nothing | 12:41 |
| spatel | reason i am asking because i made change in pki_authority here - https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ssl.yml | 12:42 |
| spatel | and re-run playbook which added RootCA in all nodes so that is good but rabbitmq still using same old cert which is ExampleCorp | 12:42 |
| spatel | i want to force rabbit to re-generate new cert and signed with MyCompanyCA | 12:42 |
| spatel | I didn't find anyway to re-generate node certs | 12:43 |
| noonedeadpunk | you would need to set `pki_regen_cert=true` while running rabbitmq role | 12:43 |
| spatel | huh.. | 12:43 |
| noonedeadpunk | but there's a bug in pki role now which should be solved with https://review.opendev.org/c/openstack/ansible-role-pki/+/808022 | 12:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_masakari master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/810221 | 12:44 |
| spatel | let me try.. this is hidden bomb :) | 12:44 |
| spatel | so it regenerate - rabbitmq_os-infra-1-rabbit-mq-container-180fd38b.crt and rabbitmq_os-infra-1-rabbit-mq-container-180fd38b.info | 12:48 |
| spatel | but no rabbitmq_os-infra-1-rabbit-mq-container-180fd38b-chain.crt | 12:48 |
| spatel | still rabbitMQ using ExampleCorp | 12:48 |
| spatel | openssl s_client -connect 172.30.40.135:5671 -cert rabbitmq.pem -key rabbitmq.key -CAfile rabbitmq-ca.pem | 12:48 |
| noonedeadpunk | but you've re-generated root right? | 12:49 |
| spatel | yes, i can see RootCA got install in every node in OSA | 12:50 |
| spatel | noonedeadpunk https://paste.opendev.org/show/809462/ | 12:51 |
| spatel | you can see I have both RootCA ExampleCorp and Vivox | 12:51 |
| noonedeadpunk | um, yes, role doesn't drop old CA from trust stores... | 12:51 |
| noonedeadpunk | that;'s true and fair | 12:52 |
| spatel | now i want rabbitMQ generate /etc/rabbitmq/rabbitmq.pem cert using new RootCA | 12:52 |
| spatel | why do we need to drop old CA.. that is ok to have it | 12:52 |
| noonedeadpunk | but it's generated based on root that is stored on the deploy host in /etc/openstack_deploy/pki/ | 12:52 |
| spatel | one interesting thing i found which is i removed RootCA from compute node but still compute node able to talk to RabbitMQ over SSL ( when you remove rootCA then it shouldn't trust right?) | 12:53 |
| noonedeadpunk | um, I guess yes | 12:55 |
| spatel | all i did remove RootCA from /etc/ssl/certs/ca-certificates.crt file from compute nodes, and restart compute nodes but still i am able to spin up VMs :) | 12:56 |
| spatel | after i removed /etc/openstack_deploy/pki/certs/certs/rabbitmq_os-infra-1-rabbit-mq-container-180fd38b-chain.crt file it work with `pki_regen_cert=true` while running rabbitmq role | 12:57 |
| spatel | i can see RabbitMQ generate node certs and signed with MyCompany RootCA | 12:58 |
| spatel | https://paste.opendev.org/show/809463/ | 12:58 |
| noonedeadpunk | I guess after reboot ca-certificates.crt got re-generated? | 12:58 |
| noonedeadpunk | based on the contents of /etc/ssl/certs/ ? | 12:59 |
| noonedeadpunk | because roots are placed as files there, and then with hook are added to ca-certificates.crt | 12:59 |
| spatel | let me give it a try to wipe out everything and see.. | 12:59 |
| spatel | one more thing why i am not seeing compute node certs in /etc/openstack_deploy/pki/ anywhere | 13:00 |
| spatel | i thought cert should get generated for each node right? | 13:00 |
| spatel | noonedeadpunk am i wrong here? | 13:06 |
| spatel | no compute nodes cert here - https://paste.opendev.org/show/809465/ | 13:07 |
| noonedeadpunk | um, no, we generate certs only for hosts/containers that needs them | 13:16 |
| noonedeadpunk | using tls for live migrations is smth not implemented yet | 13:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/810234 | 13:20 |
| spatel | so no cert for compute node right ? | 13:21 |
| noonedeadpunk | yep | 13:21 |
| spatel | good to know | 13:21 |
| noonedeadpunk | eventually we use currently only for haproxy and rabbitmq | 13:21 |
| noonedeadpunk | and now I'm pushing patches for galera | 13:21 |
| spatel | that what i can see :) | 13:21 |
| spatel | let me play with compute node cert trust and see.. | 13:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/810237 | 13:27 |
| spatel | noonedeadpunk i have successfully remove RootCA from compute node which is located here - /usr/local/share/ca-certificates | 13:28 |
| spatel | i have removed all file and now when i am running openssl client to validate its saying validation failed | 13:29 |
| spatel | but still able to spin up VM using that compute nodes so that is very odd.. | 13:29 |
| noonedeadpunk | hm... | 13:30 |
| noonedeadpunk | maybe smth is rolled back and cert verification is not required nowadays... | 13:30 |
| noonedeadpunk | weird though | 13:31 |
| spatel | look like | 13:31 |
| spatel | running this command from compute node - openssl s_client -connect 172.30.40.135:5671 | 13:32 |
| spatel | and seeing error - verify error:num=20:unable to get local issuer certificate | 13:32 |
| spatel | but rabbitMQ client is happy | 13:32 |
| noonedeadpunk | no idea really | 13:32 |
| noonedeadpunk | as I can;'t really recall now dependency that required that | 13:33 |
| spatel | This is good to know, glad we did validation | 13:34 |
| spatel | I am planning to upgrade my prod with OSA default cert this time.. :) i will change my own next time | 13:35 |
| spatel | i want to learn PKI before i push out anything case outage | 13:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_murano master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/810246 | 14:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/810247 | 14:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/810250 | 14:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_sahara master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/810252 | 14:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tacker master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/810255 | 14:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/810257 | 14:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/810258 | 14:32 |
| noonedeadpunk | damn, ironic role seem pretty broken... | 14:45 |
| noonedeadpunk | if you allow - I will skip todays meeting - feel super exhausted :( | 15:01 |
| noonedeadpunk | and don't have much to discuss, except pending reviews :) | 15:01 |
| jrosser | good idea - i'm pretty much the same | 15:02 |
| fungi | jrosser: looks like we might have another osa site pulling updates directly from each node: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66611&rra_id=all | 15:54 |
| fungi | what's the user agent string you added? | 15:54 |
| fungi | someone seems to have turned something on right at 15:30 utc anyway, we're working to figure out what/who now | 15:55 |
| jrosser | it should go something like this https://github.com/openstack/openstack-ansible-openstack_hosts/commit/f2220c4fe05ac41c512280b9be2c586acdb9ddd3 | 15:56 |
| fungi | thanks! | 15:56 |
| *** rpittau is now known as rpittau|afk | 16:25 | |
| mgariepy | https://cpaelzer.github.io/blogs/002-migration-with-changed-roms/ | 17:08 |
| mgariepy | very fun. | 17:08 |
| *** sshnaidm is now known as sshnaidm|off | 17:22 | |
| spatel | where is jamesdenton :) | 17:41 |
| spatel | we are very behind in adding new networks stuff without you :) | 17:41 |
| spatel | just realized OVN with DPDK support is broken in OSA and may need some work | 17:43 |
| jamesdenton | hi spatel | 18:25 |
| spatel | :) | 18:25 |
| jamesdenton | i'm at negative cycles :D | 18:26 |
| jamesdenton | but maybe i can find some time soon. what's up? | 18:26 |
| spatel | what is going on? | 18:26 |
| spatel | try to setup DPDK on ubuntu but having hard time | 18:26 |
| jamesdenton | OVS+DPDK? | 18:26 |
| spatel | yes | 18:26 |
| spatel | ovs-vsctl add-port br-provider dpdk-0 -- set Interface dpdk-0 type=dpdk options:dpdk-devargs=0000:06:00.1 | 18:27 |
| spatel | ovs-vsctl: Error detected while setting up 'dpdk-0': Error attaching device '0000:06:00.1' to DPDK. See ovs-vswitchd log for details. | 18:27 |
| spatel | https://paste.opendev.org/show/809478/ | 18:27 |
| spatel | do you have any idea ? | 18:28 |
| jamesdenton | htm | 18:28 |
| jamesdenton | hrm, rather | 18:28 |
| spatel | i believe you also tested DPDK with ubuntu, do you know what version of OVS did you used? | 18:29 |
| jamesdenton | not offhand, no. it would've likely been xenial, maybe bionic | 18:30 |
| jamesdenton | what do you see with "dpdk-devbind.py --status" | 18:30 |
| spatel | I did with CentOS 7.x while ago but same method not working with ubuntu | 18:30 |
| spatel | https://paste.opendev.org/show/809479/ | 18:30 |
| spatel | everything looks correct | 18:31 |
| spatel | here i documented process last year with centos - https://satishdotpatel.github.io/openstack-ansible-add-compute-node-using-openvswitch-dpdk/ | 18:31 |
| jamesdenton | Is this you? :D -- http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018517.html | 18:35 |
| jamesdenton | error looks the same, maybe there's something missing from the process? | 18:35 |
| spatel | :) | 18:37 |
| spatel | dpdk and sriov is my thing so it must be me | 18:37 |
| spatel | there is no answer on that post | 18:38 |
| jamesdenton | true - but that was on your centos 8 deploy. has it not been working at all? | 18:38 |
| jamesdenton | centos or ubuntu? | 18:39 |
| jamesdenton | i've got a 2-node OVS Ussuri deployment here, lemme see if i can follow the guide and get it going | 18:41 |
| jamesdenton | these also have ixgbe nics, but x540 | 18:41 |
| spatel | sounds good | 18:41 |
| spatel | lets validate process.. | 18:42 |
| spatel | you don't need to deploy OSA | 18:42 |
| spatel | just install openvswitch and enable dpdk | 18:42 |
| spatel | then create foo bridge and attach to DPDK interface | 18:42 |
| spatel | let me upgrade NIC driver and see if that help | 18:46 |
| jamesdenton | can you paste "ovs-vsctl show" too? | 18:48 |
| jamesdenton | i need to get my kernel setup and reboot, one sec | 18:48 |
| spatel | https://paste.opendev.org/show/809480/ | 18:49 |
| spatel | i didn't install any openstack stuff, i am just testing with lab box to attach dpdk with ovs port | 18:50 |
| spatel | not doing any OSA stuff until i figure out i can attach dpdk to ovs | 18:50 |
| jamesdenton | sure, i was just checking datapath | 18:51 |
| spatel | k | 19:01 |
| spatel | after upgrading NIC driver still same issue | 19:01 |
| spatel | jamesdenton in lsmod | grep vfio_pci is empty | 19:06 |
| spatel | look like it didn't load vfio_pci not sure that is the issue or not | 19:06 |
| spatel | jamesdenton my dmesg showing this message - vfio-pci 0000:06:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor. | 19:46 |
| spatel | look like something is wrong here | 19:46 |
| jamesdenton | that looks familiar, i may have something on that | 19:46 |
| jamesdenton | i'm working thru the issue on my side | 19:47 |
| spatel | no worry | 19:56 |
| spatel | jamesdenton i have to go but you can send me email if you find workaround | 20:14 |
| jamesdenton | i'll ping you | 20:14 |
| spatel | i may be offline in IRC :) | 20:14 |
| jamesdenton | FWIW i'm seeing the same thing. Been a while since i've messed with this | 20:14 |
| spatel | something is missing | 20:14 |
| spatel | make sure you have latest BIOS version | 20:14 |
| spatel | check this out - https://community.hpe.com/t5/ProLiant-Servers-Netservers/Disabling-RMRDS-RMRR-HP-Shared-Memory-features-on-Microserver/td-p/7105623#.YUo9qWZKjCw | 20:23 |
| spatel | gotta go | 20:32 |
| spatel | see you later | 20:32 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!