| *** lile has quit IRC | 00:10 | |
| *** cah_link has quit IRC | 00:13 | |
| *** diurnalist has quit IRC | 00:35 | |
| *** diurnalist has joined #openstack-kolla | 00:37 | |
| *** sean-k-mooney has joined #openstack-kolla | 00:39 | |
| *** k_mouza has joined #openstack-kolla | 00:45 | |
| *** k_mouza has quit IRC | 00:49 | |
| *** sean-k-mooney has quit IRC | 00:59 | |
| *** diurnalist has quit IRC | 01:11 | |
| *** diurnalist has joined #openstack-kolla | 01:18 | |
| *** diurnalist has quit IRC | 01:49 | |
| *** mrunge has quit IRC | 02:30 | |
| *** mrunge has joined #openstack-kolla | 02:32 | |
| *** zhanglong has joined #openstack-kolla | 03:13 | |
| *** zhanglong has quit IRC | 03:36 | |
| *** zhanglong has joined #openstack-kolla | 03:37 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image https://review.opendev.org/708273 | 03:48 |
|---|---|---|
| *** hongbin has joined #openstack-kolla | 03:49 | |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:15 |
| *** factor has quit IRC | 04:31 | |
| *** factor has joined #openstack-kolla | 04:31 | |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:Add support for encrypting glance/heat api https://review.opendev.org/707131 | 04:32 |
| openstackgerrit | Hongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node https://review.opendev.org/708213 | 04:34 |
| *** hongbin has quit IRC | 04:39 | |
| *** zhanglong has quit IRC | 04:53 | |
| *** zhanglong has joined #openstack-kolla | 04:55 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic https://review.opendev.org/664516 | 05:02 |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-kolla | 05:35 | |
| *** skramaja has joined #openstack-kolla | 05:35 | |
| *** zhanglong has quit IRC | 05:58 | |
| *** zhanglong has joined #openstack-kolla | 06:01 | |
| *** sri_ has joined #openstack-kolla | 06:04 | |
| *** kozhukalov has joined #openstack-kolla | 06:08 | |
| *** zhanglong has quit IRC | 06:20 | |
| *** zhanglong has joined #openstack-kolla | 06:22 | |
| *** shyamb has joined #openstack-kolla | 06:28 | |
| *** tony31 has joined #openstack-kolla | 06:37 | |
| *** k_mouza has joined #openstack-kolla | 06:46 | |
| *** k_mouza has quit IRC | 06:51 | |
| *** cah_link has joined #openstack-kolla | 07:09 | |
| *** shyamb has quit IRC | 07:14 | |
| yoctozepto | morning | 07:32 |
| sri_ | morning | 07:40 |
| sri_ | yoctozepto, sorry for silly question, how many controller we required to run 30 to 50 compute nodes with DVR enabled | 07:45 |
| yoctozepto | sri_: even one controller might be just fine, not sure if dvr changes anything in that regard; for ha you might still want more, probably 3 for default service placement | 07:49 |
| sri_ | yoctozepto, planing to use 3 controller for sure, my workload is very generic, create a bunch of vm and just running schedule snapshots and backups. | 07:52 |
| sri_ | yoctozepto, with dvr the network traffic don't need to the controllers | 07:53 |
| cosmicsound | good day | 07:54 |
| *** dciabrin has joined #openstack-kolla | 07:54 | |
| sri_ | I think 3 controller should be ok as you mentioned | 07:55 |
| yoctozepto | sri_: it never has to go there, you probably mixed controller with networking node | 07:55 |
| sri_ | yoctozepto, yes network with mixed with controller nodes | 07:56 |
| cosmicsound | yoctozepto , how can we debug better this designate deploy process in kolla, there are some weird issues here to look into, last time i had in designate_worker another pool id from the one in passwords.yml. now the pool id its good and still say no pool available | 07:59 |
| *** bengates has joined #openstack-kolla | 08:02 | |
| tony31 | morning | 08:03 |
| yoctozepto | cosmicsound: you can run services with debug=True if logs are not satisfactorily detailed | 08:07 |
| *** tonythomas has joined #openstack-kolla | 08:11 | |
| *** shyamb has joined #openstack-kolla | 08:21 | |
| cosmicsound | I have the enable_debug true | 08:22 |
| cosmicsound | isnt this same as you mentioned? | 08:22 |
| cosmicsound | yoctozepto , i found a issues | 08:24 |
| cosmicsound | In designate-sink default.conf we have this | 08:24 |
| cosmicsound | https://mdb.uhlhost.net/uploads/fce149d037300410/image.png | 08:24 |
| cosmicsound | if you check [handler:nova_fixed] / zone_id and [handler:neutron_floatingip] / zone id | 08:24 |
| cosmicsound | They are not filled in | 08:24 |
| cosmicsound | Now how do I debug this why is not filled, could it be a error in playbook? | 08:25 |
| cosmicsound | same issues we debated last time with yankcrime | 08:25 |
| mnasiadka | cosmicsound: do you see this? https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/designate/templates/designate.conf.j2#L66 | 08:34 |
| mnasiadka | we have no automation to fill in the zone-id, you need to do it by yourself in /etc/kolla/config/designate.conf | 08:34 |
| cosmicsound | its missing only 2 locations in fact in designate-sink | 08:36 |
| cosmicsound | the rest of the values are filled in | 08:37 |
| cosmicsound | it helped me last time by adding the values in designate-sink.conf | 08:37 |
| cosmicsound | and reconfigure testing it now as we speak | 08:37 |
| mnasiadka | cosmicsound: as I said, kolla-ansible will leave it blank, you need to configure it somewhere in /etc/kolla/config | 08:42 |
| cosmicsound | right | 08:51 |
| cosmicsound | editing the template also helps | 08:51 |
| cosmicsound | or il try | 08:51 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 08:52 |
| *** shyamb has quit IRC | 08:55 | |
| mgoddard | morning | 09:01 |
| tony31 | morning | 09:06 |
| *** lennyb has quit IRC | 09:06 | |
| *** lennyb has joined #openstack-kolla | 09:07 | |
| tony31 | when I run "kayobe overcloud host configure" it tries to run some variables as commands once it finishes. I only noticed this start today - I think I borked the control host 🙈 | 09:07 |
| tony31 | `(kayobe) [cv-user@juc-kach1-prd kayobe]$ hilosipip_interface: bond2bash: hilosipip_interface:: command not found` | 09:08 |
| yoctozepto | bond2bash :D | 09:08 |
| tony31 | :) | 09:08 |
| yoctozepto | Bond, Bash Bond | 09:09 |
| yoctozepto | ;-) | 09:09 |
| tony31 | usually it's bin bash | 09:09 |
| tony31 | 🗑️ | 09:09 |
| well100 | Morning | 09:15 |
| *** dougsz has joined #openstack-kolla | 09:15 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: base and openstack-base images https://review.opendev.org/709537 | 09:17 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 | 09:17 |
| well100 | mgoddard i have a doing to my patch? | 09:19 |
| *** ktibi has joined #openstack-kolla | 09:20 | |
| yoctozepto | well100: yeah, there is review/testing progress | 09:22 |
| well100 | yoctozepto: okay fine im wait for instruciton :-) | 09:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path https://review.opendev.org/709713 | 09:34 |
| hrw | well100: https://docs.openstack.org/tripleo-docs/latest/contributor/contributions.html | 09:39 |
| hrw | ops | 09:40 |
| hrw | well100: I meant https://wiki.openstack.org/wiki/How_To_Contribute | 09:40 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: CentOS 8: Enable TLS on core scenario https://review.opendev.org/709994 | 09:44 |
| mgoddard | well100: hi. Are you asking what needs to be done to your patch? | 09:45 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 09:46 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 09:46 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Testing cloudkitty in master https://review.opendev.org/681555 | 09:47 |
| yoctozepto | hrw: sending folks to tripleo? HOW DARE YOU :D | 09:49 |
| mgoddard | well100: yoctozepto has posted some comments on your patch | 09:50 |
| mgoddard | (assume we are talking about https://review.opendev.org/#/c/707379) | 09:50 |
| patchbot | patch 707379 - kolla-ansible - Cloudkitty cant not conncet to Auth - 2 patch sets | 09:50 |
| hrw | can't not connect? | 09:53 |
| well100 | mgoddard: should i make a new patch without v3? | 10:04 |
| openstackgerrit | Alfredo Moralejo proposed openstack/kolla master: Use StorageSIG repos for Ceph in CentOS8 https://review.opendev.org/707338 | 10:06 |
| openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy https://review.opendev.org/709580 | 10:08 |
| openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Allow setting pip_proxy https://review.opendev.org/709580 | 10:10 |
| tony31 | hi kayobe guys | 10:11 |
| tony31 | I managed to trace back this problem which I am having at the moment. I am using one physical node for 2 roles: compute and storage. I found that the tunnel network is not being set on the compute node any longer. Not sure how long this issue has been there. But if I remove the storage role from this node then it successfully gets the tunnel | 10:12 |
| tony31 | network | 10:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 10:12 |
| mgoddard | hi tony31, I see. It is normally best to only have one top level role per node | 10:13 |
| mgoddard | due to the way we build our lists of network interfaces etc | 10:13 |
| tony31 | fair enough :) | 10:13 |
| mgoddard | I'd suggest using compute, and adding storage bits as necessary | 10:13 |
| tony31 | trying to make the most of the hardware I have - I'll try and spin up a storage VM on another hypervisor | 10:14 |
| tony31 | thanks for the advice on it | 10:14 |
| mgoddard | tony31: you could add the compute group to kolla_overcloud_inventory_storage_groups | 10:15 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 10:15 |
| tony31 | hmm | 10:15 |
| mgoddard | and add networks to compute_extra_network_interfaces as necessary | 10:15 |
| tony31 | is that only in the venv ? | 10:16 |
| tony31 | `$ grep -rni -e "kolla_overcloud_inventory_storage_groups"venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:275:kolla_overcloud_inventory_storage_groups:venvs/kayobe/share/kayobe/ansible/group_vars/all/kolla:298: "{{ kolla_overcloud_inventory_storage_groups }}"` | 10:16 |
| mgoddard | tony31: we don't include it in etc/kayobe/kolla.yml but it seems to be required quite frequently so maybe we should add it | 10:17 |
| mgoddard | tony31: but in general you can add arbitrary variables to the files in etc/kayobe/*.yml | 10:17 |
| mgoddard | we just include the most common ones | 10:17 |
| tony31 | Would it work the other way around? if I add `storage_extra_network_interfaces:` and list the tunnel network there? | 10:18 |
| *** hjensas has quit IRC | 10:18 | |
| *** k_mouza has joined #openstack-kolla | 10:21 | |
| *** hjensas has joined #openstack-kolla | 10:21 | |
| *** k_mouza has quit IRC | 10:21 | |
| *** k_mouza has joined #openstack-kolla | 10:21 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: DNM: Test swift s3api https://review.opendev.org/709730 | 10:24 |
| tony31 | mgoddard - I added `storage_extra_network_interfaces:` and set the tunnel interface there (in a host_vars) and now when I do a host configure, it is including the tunnel network. This is just for testing at the moment. I'm still pulling the pieces together. | 10:27 |
| tony31 | i just needed a storage node in the same area that the storage is located so that the build is fast - iscsi over a VPN tunnel was a bit slow (for glance building instances) | 10:28 |
| tony31 | :) | 10:28 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 10:29 |
| *** skramaja has quit IRC | 10:30 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients https://review.opendev.org/709751 | 10:34 |
| *** tony31 has quit IRC | 10:35 | |
| hrw | bbl | 10:37 |
| openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:41 |
| openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:41 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 10:41 |
| openstackgerrit | yatin proposed openstack/kolla master: Build collectd image for CentOS8 https://review.opendev.org/709701 | 10:42 |
| *** sean-k-mooney has joined #openstack-kolla | 10:45 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 10:52 |
| openstackgerrit | Merged openstack/kolla stable/train: vitrage-base: honor distro_python_version https://review.opendev.org/709553 | 10:57 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix RabbitMQ hostname address resolution precheck https://review.opendev.org/707892 | 10:57 |
| *** priteau has joined #openstack-kolla | 11:01 | |
| well100 | i have question to rdns :-) | 11:09 |
| well100 | how do i create the dns zone for it to work | 11:10 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Allow to override external network params in init-runonce https://review.opendev.org/709671 | 11:21 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Fix TLS upgrade test https://review.opendev.org/709808 | 11:22 |
| *** shyamb has joined #openstack-kolla | 11:27 | |
| *** shyamb has quit IRC | 11:45 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor https://review.opendev.org/709203 | 11:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master https://review.opendev.org/709204 | 11:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh https://review.opendev.org/709205 | 11:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location https://review.opendev.org/709207 | 11:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: Switch to python3 in bindep.txt https://review.opendev.org/709757 | 11:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Add deploy jobs in CI https://review.opendev.org/709538 | 11:59 |
| mgoddard | yoctozepto, mnasiadka, osmanlicilegi: https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train | 12:00 |
| *** shyamb has joined #openstack-kolla | 12:08 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 12:22 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: Add addressing on external network https://review.opendev.org/709362 | 12:27 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for monasca agent CA file https://review.opendev.org/709711 | 12:31 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Remove unused python path calculation from vmtp https://review.opendev.org/709712 | 12:31 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Use local python interpreter for keystone cron generator https://review.opendev.org/709714 | 12:31 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 12:34 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Support python 3 in kolla-ansible script https://review.opendev.org/709716 | 12:36 |
| ktibi | Hello guys, I add a param in kayobe for the registry password. But I can see in the password.yml of kolla, the password is set to null. How can I fix that ? Need to edit manualy the password.yml ? | 12:36 |
| ktibi | or maybe I can regenerate the file :/ | 12:39 |
| mgoddard | ktibi: I'd suggest using this: https://docs.openstack.org/kayobe/latest/configuration/kolla-ansible.html#configuring-custom-passwords | 12:43 |
| mgoddard | maybe we need to add that to the defaults? | 12:43 |
| ktibi | ok strange because I have set docker_registry_password (I can see it in global of kolla) but when kayobe check if passwords.yml is correct, kayobe override and add None to the password | 12:44 |
| ktibi | mgoddard, ok because kayobe use "{{ temp_path }}/passwords.yml" and I think I need to modify this file :p | 12:46 |
| ktibi | ok found, need ot modify etc/kayobe/kolla/password ;) | 12:49 |
| *** kplant has joined #openstack-kolla | 12:57 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 12:58 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros https://review.opendev.org/710027 | 12:58 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 12:59 |
| openstackgerrit | Michal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed https://review.opendev.org/709689 | 13:01 |
| openstackgerrit | Piotr Rabiega proposed openstack/kolla master: Add collectd-pcie-errors package for PCIe Errors (C7 only) https://review.opendev.org/710029 | 13:06 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 13:06 |
| well100 | ptr reverse dns can you help me? | 13:07 |
| cosmicsound | well100 , what is wrong | 13:10 |
| cosmicsound | i work also now on designate | 13:10 |
| well100 | do I have to create the ptr zone, if so where do I have to create it? | 13:11 |
| cosmicsound | you can do it in /etc/hosts locally or with you provider, not sure how your domain is routed | 13:13 |
| cosmicsound | i use for example a dns in cpanel now, that i forward to my server where i make the reverse dns | 13:13 |
| cosmicsound | and then i add it locally in /etc/hosts | 13:13 |
| cosmicsound | to reflect the ip of the main bind9 host in designate | 13:14 |
| cosmicsound | in my case that is dns.uhlhost.net if you do a host dns.uhlhost.net you see it points to my designate ip for this. | 13:14 |
| cosmicsound | not sure if this helps | 13:14 |
| cosmicsound | also if you have more ips your provider should allow you to change the reverse records, once ofcourse a dns points to them | 13:15 |
| cosmicsound | or maybe using the bind templates | 13:15 |
| cosmicsound | you could change there all ptr records | 13:15 |
| *** skramaja has joined #openstack-kolla | 13:16 | |
| well100 | no | 13:26 |
| well100 | I want designate to create the PTR records of fixed and floating ips | 13:27 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Python 3: Use distro_python_version for WSGI python_path https://review.opendev.org/709713 | 13:35 |
| osmanlicilegi | mnasiadka: after merging https://review.opendev.org/#/c/707375/ I've discovered ubuntu cannot manage network namespaces anymore. should be same on debian too. | 13:38 |
| patchbot | patch 707375 - kolla-ansible - Change /run bind mount for neutron/openvswitch (MERGED) - 4 patch sets | 13:38 |
| osmanlicilegi | seems /run rules ubuntu | 13:38 |
| osmanlicilegi | I'll raise a bug report soon | 13:39 |
| yoctozepto | osmanlicilegi: dang | 13:41 |
| yoctozepto | osmanlicilegi: that's odd because it is passing in CI | 13:42 |
| yoctozepto | wonder which part broke there | 13:42 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: WIP:support for separate admin vip address and admin fqdn https://review.opendev.org/710036 | 13:42 |
| *** diurnalist has joined #openstack-kolla | 13:44 | |
| openstackgerrit | Merged openstack/kolla stable/train: Remove tgtd and scsi-target-utils support in CentOS/RHEL 8 https://review.opendev.org/702421 | 13:48 |
| *** diurnalist has quit IRC | 13:48 | |
| mgoddard | :( ubuntu deploy jobs seem broken on master | 13:50 |
| *** shyamb has quit IRC | 13:51 | |
| yoctozepto | where? how? | 13:53 |
| mgoddard | possible it's caused by dropping epel :) | 13:54 |
| mgoddard | it's probably fine | 13:54 |
| mgoddard | hold up that doesn't make sense on ubuntu! | 13:55 |
| mgoddard | wake up mgoddard | 13:55 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Actually disable EPEL, and epel-modular https://review.opendev.org/710003 | 13:59 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Throw TypeError in repository enable/disable macros https://review.opendev.org/710027 | 13:59 |
| kevinz | yoctozepto, morning | 14:04 |
| kevinz | do we have a procedure to create octavia managment network? | 14:05 |
| *** zhanglong has quit IRC | 14:06 | |
| *** zhanglong has joined #openstack-kolla | 14:07 | |
| *** dasp has quit IRC | 14:08 | |
| mnasiadka | osmanlicilegi: that's weird, ubuntu is special with netns or what? :) | 14:10 |
| mnasiadka | osmanlicilegi: if you can just paste neutron logs somewhere - we can analyse | 14:11 |
| *** zhanglong has quit IRC | 14:14 | |
| *** zhanglong has joined #openstack-kolla | 14:16 | |
| osmanlicilegi | mnasiadka: last 500 lines of neutron-l3-agent https://paste.ubuntu.com/p/D6PfpWZSvj/ | 14:20 |
| osmanlicilegi | I've discovered that all namespaces were lost. after rolling back that change, all came back. | 14:21 |
| mnasiadka | osmanlicilegi: so that means we need to add /run/netns mount | 14:21 |
| mnasiadka | osmanlicilegi: do you have an env to check, if changing /run/ bindmount to /run/netns helps? | 14:22 |
| mnasiadka | well, helps... doesn't break anything :D | 14:24 |
| osmanlicilegi | mnasiadka: I'll test it, give me some time | 14:25 |
| *** sri_ has quit IRC | 14:25 | |
| mnasiadka | osmanlicilegi: I just can't wait to do a revert on each branch... | 14:25 |
| openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs https://review.opendev.org/702452 | 14:26 |
| *** abdysn has joined #openstack-kolla | 14:28 | |
| *** abdysn has quit IRC | 14:28 | |
| mnasiadka | osmanlicilegi: but that's really interesting centos doesn't have a problem with that, nor Ubuntu in CI had a problem | 14:28 |
| mnasiadka | osmanlicilegi: just checked CI jobs result on Ubuntu, worked like charm | 14:33 |
| *** zhanglong has quit IRC | 14:38 | |
| *** sean-k-mooney has quit IRC | 14:38 | |
| *** kplant has quit IRC | 14:43 | |
| *** kplant has joined #openstack-kolla | 14:43 | |
| mnasiadka | yoctozepto: I think it may be related to update of existing routers in neutron | 14:44 |
| osmanlicilegi | mnasiadka: reconfigure on the way... | 14:45 |
| mgoddard | osmanlicilegi: do the netns disappear after restarting the container? | 14:51 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 14:51 |
| mnasiadka | mgoddard: currently we only know neutron has problems in accessing netns, in router_update and router_delete | 14:52 |
| mnasiadka | mgoddard: and without bindmounting /run/netns to the host, they might be not persistent :) | 14:52 |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi | 14:55 |
| mgoddard | meeting in 5 ^ | 14:55 |
| yoctozepto | 4 | 14:56 |
| yoctozepto | :D | 14:56 |
| mgoddard | slowest countdown ever | 14:56 |
| mgoddard | mnasiadka: tripleo also mounts /run/netns | 14:56 |
| yoctozepto | mgoddard: never counted days to see your loved one, did you? | 14:56 |
| osmanlicilegi | mnasiadka: seems mounting /run/netns fixes | 14:56 |
| mgoddard | also /lib/modules:/lib/modules:ro | 14:57 |
| mnasiadka | that one I have no clue why :) | 14:57 |
| mgoddard | also looked like they have to create a dummy netns to ensure the directory exists | 14:57 |
| yoctozepto | mnasiadka: weird, it should hit CI hard | 14:57 |
| yoctozepto | hmm | 14:58 |
| mnasiadka | yoctozepto: well, it should on reconfigure, but it's not hitting | 14:58 |
| *** jbadiapa has quit IRC | 14:58 | |
| mnasiadka | unless we are not restarting neutron on reconfigure | 14:58 |
| yoctozepto | we are not | 14:58 |
| yoctozepto | so it must be restarted to fail? | 14:58 |
| mgoddard | ovn metadata agent also has those mounts | 14:58 |
| mnasiadka | maybe we should force reconfigure to restart all docker containers :D | 14:58 |
| yoctozepto | then upgrades should be hit anyway | 14:58 |
| mnasiadka | mgoddard: yeah, metadata agent also needs access to netns | 14:59 |
| osmanlicilegi | https://bugs.launchpad.net/kolla-ansible/+bug/1864856 | 14:59 |
| openstack | Launchpad bug 1864856 in kolla-ansible "dhcp-agent failed to open netns" [Undecided,New] | 14:59 |
| hrw | mgoddard: thanks | 14:59 |
| osmanlicilegi | same issue | 14:59 |
| hrw | time to start | 15:00 |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Feb 26 15:01:04 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| mgoddard | #topic rollcall | 15:01 |
| openstack | The meeting name has been set to 'kolla' | 15:01 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| mgoddard | \o | 15:01 |
| osmanlicilegi | o/ | 15:01 |
| yoctozepto | o/ | 15:01 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 15:01 |
| hrw | /o/ | 15:01 |
| *** TrevorV has joined #openstack-kolla | 15:01 | |
| mnasiadka | o/ | 15:01 |
| hrw | \o\ /°\ | 15:01 |
| osmanlicilegi | \o/\o/\o/ | 15:02 |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:03 |
| mgoddard | * Announcements | 15:03 |
| mgoddard | * Review action items from last meeting | 15:03 |
| mgoddard | * CI status | 15:03 |
| mgoddard | * Ussuri release planning (kolla & kolla ansible) | 15:03 |
| mgoddard | * Ussuri release planning (kayobe) | 15:03 |
| mgoddard | * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig | 15:03 |
| mgoddard | #topic announcements | 15:03 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:03 | |
| mgoddard | #info Rocky will move to extended maintenance (EM) soon | 15:04 |
| hrw | 24.02 was a date iirc | 15:04 |
| mgoddard | Waiting for final rocky releases then we can bump versions and release our own final | 15:04 |
| mgoddard | Any other announcements? | 15:05 |
| yoctozepto | we finally tested to-instance network connectivity in CI | 15:05 |
| yoctozepto | though that did not catch netns failure ;D | 15:05 |
| mgoddard | yeah that's nice | 15:06 |
| *** lile has joined #openstack-kolla | 15:06 | |
| mgoddard | #topic Review action items from last meeting | 15:06 |
| *** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:06 | |
| osmanlicilegi | yoctozepto: if ci catches everything, we couldn't have adventure :] | 15:06 |
| mgoddard | mnasiadka request neutron 14.1.0 in stein UCA | 15:06 |
| mgoddard | yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 | 15:06 |
| mgoddard | jovial[m] to work on custom extension points | 15:06 |
| mgoddard | dougsz to write bug report about nova SSH nproc issue | 15:07 |
| mgoddard | mnasiadka: done? | 15:07 |
| mnasiadka | mgoddard: complained, but they said they have their own testing regime and will take some time | 15:07 |
| yoctozepto | regime | 15:07 |
| yoctozepto | totalitarian I presume | 15:07 |
| mnasiadka | whatever that means | 15:07 |
| mgoddard | ok | 15:07 |
| mgoddard | drop ubuntu | 15:08 |
| yoctozepto | RIP Stein CI | 15:08 |
| mgoddard | yoctozepto: done? | 15:08 |
| mgoddard | yes | 15:08 |
| mgoddard | thanks | 15:08 |
| yoctozepto | yw | 15:08 |
| mgoddard | jovial[m] is away, probably more of a long term thing | 15:08 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 15:09 |
| mgoddard | dougsz: you wrote the bug report didn't you? | 15:09 |
| dougsz | ah sorry, still on my todo list, will do it today | 15:10 |
| mgoddard | #action dougsz to write bug report about nova SSH nproc issue | 15:10 |
| mgoddard | #topic CI status | 15:10 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:11 | |
| *** TrevorV has quit IRC | 15:11 | |
| mgoddard | Looks green apart from stein ubuntu-binary | 15:11 |
| *** TrevorV has joined #openstack-kolla | 15:11 | |
| yoctozepto | indeed | 15:11 |
| mgoddard | I saw some weirdness with ubuntu+keystone on master earlier, hopefully it's my patch | 15:11 |
| mgoddard | although I doubt it | 15:12 |
| mgoddard | #topic Ussuri release planning (kolla & kolla ansible) | 15:12 |
| *** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)" | 15:12 | |
| mgoddard | I've been working on the centos8 train backport | 15:12 |
| yoctozepto | mgoddard: all the more reason to depreacate ubuntu | 15:12 |
| mgoddard | #link https://review.opendev.org/#/q/topic:bp/centos-rhel-8+status:open+branch:stable/train | 15:13 |
| mgoddard | thanks for all reviews so far | 15:13 |
| mgoddard | lots of approvals | 15:13 |
| mgoddard | last few: | 15:13 |
| mgoddard | https://review.opendev.org/709757 | 15:13 |
| patchbot | patch 709757 - kolla (stable/train) - Switch to python3 in bindep.txt - 3 patch sets | 15:13 |
| mgoddard | https://review.opendev.org/709204 | 15:14 |
| patchbot | patch 709204 - kolla (stable/train) - CentOS 8: Use upstream Ceph/master - 4 patch sets | 15:14 |
| mgoddard | https://review.opendev.org/709203 | 15:14 |
| patchbot | patch 709203 - kolla (stable/train) - CentOS 8: Remove shellinabox from ironic-conductor - 4 patch sets | 15:14 |
| *** sean-k-mooney has joined #openstack-kolla | 15:14 | |
| mgoddard | https://review.opendev.org/709718 | 15:14 |
| patchbot | patch 709718 - kolla-ansible (stable/train) - CI: Use python 3 for local kolla-ansible execution - 2 patch sets | 15:14 |
| mgoddard | https://review.opendev.org/709717 | 15:14 |
| patchbot | patch 709717 - kolla-ansible (stable/train) - CI: Move ansible installation & configuration to A... - 1 patch set | 15:14 |
| mgoddard | those last two need to merge before deploy jobs will pass - I was too lazy to add depends-on | 15:15 |
| mgoddard | and this one will fix ironic jobs: https://review.opendev.org/709751 | 15:16 |
| patchbot | patch 709751 - kolla-ansible (stable/train) - CI: Use upper constraints when installing clients - 2 patch sets | 15:16 |
| * mgoddard stops begging for reviews | 15:16 | |
| mnasiadka | lol | 15:16 |
| mgoddard | What other nice ussuri work should we discuss today? | 15:16 |
| mnasiadka | I spoke with CentOS Storage SIG, it seems in Ussuri we will use Nautilus | 15:17 |
| mnasiadka | Which is in a bit of contrary to what Sage said about CentOS 8 and Ceph release support | 15:17 |
| mnasiadka | Will investigate that topic, since upstream Ceph repo doesn't have Nautilus on el8 | 15:18 |
| *** skramaja has quit IRC | 15:18 | |
| mgoddard | not sage adivce | 15:18 |
| mgoddard | *advice | 15:18 |
| mnasiadka | and CentOS Storage SIG seems to have it working | 15:18 |
| mnasiadka | quite a nice desync in one company :) | 15:18 |
| hrw | Octopus was not released yet iirc | 15:18 |
| hrw | mnasiadka: which company you mean? | 15:18 |
| mnasiadka | hrw: Red Hat | 15:19 |
| hrw | mnasiadka: Ceph is not RH product | 15:19 |
| mgoddard | they probably realised someone might want to run ceph on centos 8 before the middle of this year | 15:19 |
| yoctozepto | probably | 15:20 |
| mnasiadka | mgoddard: yeah, but still those packages are from CentOS Storage SIG, not Ceph upstream - but those were always built with different deps | 15:20 |
| mgoddard | we have a patch to switch to nautilus, seems to work | 15:20 |
| mnasiadka | Now that we don't have ceph-kolla, life should be easier - whatever the release of Ceph we are using | 15:21 |
| yoctozepto | mnasiadka: a bit | 15:21 |
| yoctozepto | still need to cater for right client libs | 15:21 |
| yoctozepto | mgoddard: reviewed | 15:22 |
| mnasiadka | ok, end of Ceph topic - I just want to make sure no weird bugs will happen and Ceph bug scrubbing team will tell "we don't support this" | 15:22 |
| mnasiadka | ;-) | 15:22 |
| mgoddard | well we can't release with master | 15:23 |
| mgoddard | we can bump to octopus when available if we choose | 15:23 |
| mnasiadka | well, Ussuri release is 13th May, Octopus release is 31st March | 15:24 |
| mgoddard | generalfuzz: you around? | 15:24 |
| generalfuzz | yes | 15:25 |
| mgoddard | how is the backend tls work going? | 15:25 |
| generalfuzz | I believe it is progressing nicely. I would like to get another set of reviews on the current patch - https://review.opendev.org/#/c/664516 | 15:26 |
| patchbot | patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets | 15:26 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 15:27 |
| mgoddard | are you and yongjun bai communicating? | 15:27 |
| generalfuzz | some | 15:28 |
| mgoddard | I don't know if you've seen the patches coming in from them | 15:28 |
| mgoddard | just want to make sure everyone's on the same page | 15:28 |
| mgoddard | we've talked about trying to create common roles for some of this stuff to reduce duplication | 15:29 |
| generalfuzz | I will send a note today. My goal is to have an agreed upon implementation in https://review.opendev.org/#/c/664516. Then we can split out the services | 15:29 |
| patchbot | patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 20 patch sets | 15:29 |
| mgoddard | makes sense | 15:29 |
| generalfuzz | I'm going to look into a wsgi template generation task next | 15:29 |
| mgoddard | ok | 15:30 |
| mgoddard | quick poll: is a generic wsgi/apache config template worth doing? | 15:30 |
| osmanlicilegi | +1 | 15:30 |
| mnasiadka | I thought about the same today | 15:30 |
| yoctozepto | are we doing deprecated mod_wsgi now? | 15:31 |
| mnasiadka | mod_wsgi is deprecated? | 15:31 |
| yoctozepto | mnasiadka: some os services marked it not recommended | 15:31 |
| mnasiadka | yoctozepto: and what is recommended? | 15:31 |
| yoctozepto | mnasiadka: uwsgi is the way forward | 15:31 |
| osmanlicilegi | uwsgi I think | 15:31 |
| yoctozepto | it actually works in devstack | 15:31 |
| yoctozepto | ;p | 15:32 |
| generalfuzz | I was unable to get uwsgi to work with certs | 15:32 |
| yoctozepto | hmm | 15:32 |
| mnasiadka | yoctozepto: devstack is a buzzword, me don't believe | 15:32 |
| mgoddard | and this wasn't mentioned because... | 15:32 |
| yoctozepto | mnasiadka: trust me, I'm core ;p | 15:32 |
| mnasiadka | yoctozepto: maybe I'm old fashioned, but can OpenStack make a statement on what is the direction? because I feel in next release they will say uwsgi is bad, and we should go to this shiny new tool | 15:33 |
| mgoddard | should we be using uwsgi instead then? | 15:33 |
| yoctozepto | generally yes, except for glance which wants to stay with its old eventlet | 15:33 |
| mgoddard | ok, we have two separate things here | 15:34 |
| mnasiadka | yoctozepto: and we have a change for glance to use mod_wsgi | 15:34 |
| generalfuzz | I got exceptions when I configured services with uwsgi and defined the certs. I can revisit that today to pinpoint the exceptions | 15:34 |
| yoctozepto | mnasiadka: block it | 15:35 |
| mgoddard | 1. backend tls - general pattern, usage, etc | 15:35 |
| mgoddard | 2. backend web server | 15:35 |
| mgoddard | if 2 is contentious we can continue with 1 | 15:35 |
| yoctozepto | apache can do mod_proxy | 15:36 |
| mgoddard | but let's not go adding mod_wsgi everywhere if its genuinely deprecated | 15:36 |
| yoctozepto | generalfuzz: what broke with tls in uwsgi? | 15:36 |
| mnasiadka | https://governance.openstack.org/tc/goals/selected/pike/deploy-api-in-wsgi.html#uwsgi-vs-mod-wsgi | 15:36 |
| mnasiadka | (it's pike - but states devstack done move to uwsgi) | 15:36 |
| mgoddard | I'm fairly sure OSA supports uwsgi | 15:36 |
| yoctozepto | "with the intent that the mod_wsgi support is deleted from devstack in Queens." | 15:36 |
| openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: base and openstack-base images https://review.opendev.org/709537 | 15:36 |
| mgoddard | #link https://docs.openstack.org/ansible-role-uwsgi/latest/ | 15:37 |
| yoctozepto | yeah, osa is uwsgi | 15:37 |
| yoctozepto | or at least to some degree | 15:37 |
| mgoddard | not in devstack doesn't mean deprecated | 15:37 |
| generalfuzz | yoctozepto: there were python openssl exceptions. I will revisit today | 15:38 |
| mgoddard | generalfuzz: would be interesting to see what you changed to get uwsgi going | 15:39 |
| yoctozepto | mgoddard: right but if osa and devstack push towards uwsgi, then mod_wsgi becomes obsolete and can break | 15:40 |
| mgoddard | what about tripleo? | 15:40 |
| yoctozepto | also see: https://bugs.launchpad.net/neutron/+bug/1864418 | 15:41 |
| openstack | Launchpad bug 1864418 in neutron "has wrong with use apache to start neutron api in docker container" [Undecided,New] | 15:41 |
| yoctozepto | this might be neutron behind mod_wsgi issue | 15:41 |
| yoctozepto | mgoddard: good question | 15:41 |
| openstackgerrit | Merged openstack/kolla stable/train: CentOS 8: Update packages in images https://review.opendev.org/709202 | 15:42 |
| mgoddard | looks like quite a lot of wsgi in tripleo | 15:42 |
| mgoddard | which means kolla images probably need to keep apache packages | 15:42 |
| mgoddard | (unless we get them to override) | 15:42 |
| generalfuzz | mgoddard: uwsgi is supported out of the box for nova. I'll need to look at Placement + keystone | 15:43 |
| mgoddard | ok, sounds like more research required on wsgi vs. uwsgi | 15:43 |
| yoctozepto | generalfuzz: keystone is uwsgi-only in osa | 15:43 |
| mnasiadka | mgoddard: well, in theory we could support both | 15:43 |
| yoctozepto | mhm, tripleo looks mod_wsgi-only | 15:43 |
| generalfuzz | we may need a hybrid approach | 15:44 |
| mgoddard | mnasiadka: no thanks :) | 15:44 |
| yoctozepto | mgoddard: mnasiadka is right | 15:44 |
| yoctozepto | stuffing both required parts in kolla is nobrainer | 15:44 |
| mnasiadka | mgoddard: I think it will end up like this unfortunately | 15:44 |
| mgoddard | why? | 15:44 |
| yoctozepto | and we might want a phasing out approach | 15:44 |
| yoctozepto | ^ | 15:44 |
| yoctozepto | as it may break any day | 15:44 |
| mnasiadka | for now it works I guess, so it's not critical ;) | 15:45 |
| mgoddard | indeed | 15:45 |
| yoctozepto | agreed | 15:45 |
| mnasiadka | more convenient would be to have some common template or role that unifies mod_wsgi configs | 15:45 |
| mgoddard | generalfuzz: I would suggest not adding any more mod_wsgi configs for now :) | 15:45 |
| generalfuzz | I will re-look at uwsgi with TLS. | 15:46 |
| yoctozepto | well, those two kinda contradict each other | 15:46 |
| mgoddard | well no point in a common mod_wsgi role if we move to uwsgi | 15:46 |
| *** Trevor_V has joined #openstack-kolla | 15:46 | |
| mgoddard | screw wsgi, I need a whiskey | 15:46 |
| generalfuzz | Is adding ability to execute the container with the "root" user an acceptable solution? | 15:47 |
| mgoddard | let's try to get backend tls polished and merged with just keystone support | 15:47 |
| yoctozepto | wsgiey | 15:47 |
| generalfuzz | mgoddard: I will remove the nova + placement for now | 15:47 |
| yoctozepto | it seriously should not be necessary | 15:47 |
| mnasiadka | around uwsgi - I just hope uwsgi version between distro is at least a bit consistent, looking at mod_wsgi versions that we have now (and have to use medieval configs due to CentOS) | 15:48 |
| generalfuzz | yoctozepto: how can I have the container run the apache script with sudo from k-a code? | 15:48 |
| mgoddard | on the root user - normally we change it in the container image | 15:48 |
| mgoddard | USER root | 15:49 |
| yoctozepto | ^ | 15:49 |
| mgoddard | question is whether this presents a transition problem for tripleo or k-a | 15:49 |
| *** TrevorV has quit IRC | 15:49 | |
| generalfuzz | so I should go into docker scripts in kolla as a related change? | 15:50 |
| mgoddard | looks like tripleo might already use wsgi for now | 15:50 |
| mgoddard | yes | 15:50 |
| yoctozepto | or eventlet | 15:50 |
| yoctozepto | hard to catch that | 15:50 |
| mgoddard | I'd be interested to see what files the uwsgi config requires for TLS | 15:50 |
| yoctozepto | as it's just running py script | 15:50 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix https://review.opendev.org/709534 | 15:50 |
| mnasiadka | this looks like the easiest thing on earth: https://uwsgi-docs.readthedocs.io/en/latest/HTTPS.html | 15:51 |
| mgoddard | i.e. if we merge backend tls support for wsgi, could we switch to uwsgi without any change in the user interface (config files) | 15:51 |
| mnasiadka | So I'm pretty sure it doesn't work | 15:51 |
| *** dasp has joined #openstack-kolla | 15:52 | |
| yoctozepto | --https 0.0.0.0:8443,foobar.crt,foobar.key | 15:52 |
| yoctozepto | well, that pretty much explains what tls really is | 15:52 |
| generalfuzz | mgoddard: config files would change, since we would declare cert/key in .conf file | 15:53 |
| hrw | mgoddard: on my server I use nginx to wrap uwsgi with tls | 15:53 |
| mgoddard | generalfuzz: that's fine - we control those | 15:53 |
| mgoddard | looks like it's just a key and cert, same as wsgi | 15:54 |
| yoctozepto | we always needs this at min | 15:55 |
| yoctozepto | and then any shim to convert/glue to the required form would be sufficient | 15:55 |
| yoctozepto | like that ugly haproxy habing key with cert | 15:55 |
| yoctozepto | who came up with that | 15:55 |
| mgoddard | does devstack use mod_uwsgi or uwsgi binary? | 15:56 |
| ktibi | mgoddard, kayobe can auto resize lvm thinpool on seed if the disk is increase ? | 15:56 |
| mgoddard | ktibi: meeting time | 15:56 |
| mgoddard | ok, seems this has taken most of the meeting | 15:57 |
| mgoddard | #topic Ussuri release planning (kayobe) | 15:57 |
| *** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)" | 15:57 | |
| yoctozepto | mgoddard: uwsgi | 15:57 |
| mnasiadka | kayobe as wsgi app? | 15:57 |
| yoctozepto | mgoddard: external binary | 15:57 |
| mnasiadka | (just laughing to continue wsgi topic) | 15:58 |
| yoctozepto | mnasiadka: yes, wsgify kayobe | 15:58 |
| mgoddard | I don't have much to say other than I have more CentOS 8 patches for kayobe - reviews please dougsz & priteau :) | 15:58 |
| mgoddard | uwsgi yoctozepto mnasiadka | 15:58 |
| dougsz | :) | 15:58 |
| priteau | will do | 15:58 |
| mgoddard | thanks | 15:58 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions https://review.opendev.org/710067 | 15:59 |
| * yoctozepto cannot be deployed as wsgi app under mnasiadka | 15:59 | |
| mgoddard | I put together a testing checklist for kayobe & centos 8 | 15:59 |
| mgoddard | https://etherpad.openstack.org/p/kolla-centos8 | 15:59 |
| mgoddard | please add to it if you think of anything else | 15:59 |
| *** TrevorV has joined #openstack-kolla | 15:59 | |
| mgoddard | and if you have any time for testing please let me know | 16:00 |
| mgoddard | (no doubt testing & fixing) | 16:00 |
| mgoddard | we outta time | 16:00 |
| mgoddard | thanks all | 16:00 |
| *** diurnalist has joined #openstack-kolla | 16:00 | |
| mgoddard | #endmeeting | 16:00 |
| *** openstack changes topic to "Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:00 | |
| openstack | Meeting ended Wed Feb 26 16:00:40 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.html | 16:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.txt | 16:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-02-26-15.01.log.html | 16:00 |
| yoctozepto | thanks mgoddard | 16:00 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/stein: Bump stein versions https://review.opendev.org/710068 | 16:01 |
| hrw | for rocky we would need to do by hand checks as rocky-em is already on for some components | 16:02 |
| *** Trevor_V has quit IRC | 16:03 | |
| mgoddard | hrw: should we update the script to ignore that tag? | 16:03 |
| mgoddard | ktibi: I have resized thinpools before, not sure about automatically by kayobe though | 16:04 |
| ktibi | mgoddard, ok. I try to use --wipe-disks but kayobe don't remove lvm partition | 16:05 |
| hrw | mgoddard: on all branches probably | 16:06 |
| *** lile has quit IRC | 16:11 | |
| priteau | ktibi: --wipe-disks should clear out all LVM info, have you checked if something on that disk is still mounted? | 16:12 |
| hrw | mgoddard: https://paste.centos.org/view/26c75996 is quick change (rocky branch) | 16:12 |
| ktibi | priteau, it's on seed VM and bifrost container is not created. | 16:13 |
| *** lile has joined #openstack-kolla | 16:15 | |
| hrw | mgoddard: master has -em ignoring. we just not cherrypicked it | 16:17 |
| mgoddard | ah | 16:17 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CentOS 8: Deploy CentOS 8 containers https://review.opendev.org/709535 | 16:17 |
| hrw | guess who wrote it... | 16:17 |
| hrw | commit f528ad81026526af02610aa3e08200fd53b87ab9 | 16:17 |
| hrw | Author: Mark Goddard <mark@stackhpc.com> | 16:17 |
| hrw | Date: Mon May 20 19:00:33 2019 +0100 | 16:17 |
| hrw | Ignore EM releases in version-check.py | 16:17 |
| cosmicsound | i enable tls and define own tls in place, and in end i still get a self generate ssl how is this even possible? | 16:18 |
| yoctozepto | spoiler alert was missed | 16:18 |
| cosmicsound | or is this normal behaviour? | 16:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/stein: Ignore EM releases in version-check.py https://review.opendev.org/710073 | 16:18 |
| yoctozepto | no, it's not | 16:18 |
| yoctozepto | I run external tls and it works fine | 16:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/rocky: Ignore EM releases in version-check.py https://review.opendev.org/710074 | 16:18 |
| cosmicsound | what vars you use | 16:18 |
| cosmicsound | il share mine nnow | 16:18 |
| cosmicsound | second | 16:18 |
| cosmicsound | https://mdb.uhlhost.net/uploads/928d63db84b9ec23/image.png here are globals.yml | 16:19 |
| hrw | marked Rocky one as RP+1 | 16:19 |
| cosmicsound | haproxy-ca.crt haproxy-ca-internal.crt haproxy-internal.pem haproxy.pem have these in my /etc/kolla/certificates | 16:20 |
| yoctozepto | cosmicsound | 16:21 |
| yoctozepto | http://paste.openstack.org/show/790038/ | 16:21 |
| yoctozepto | http://paste.openstack.org/show/790039/ | 16:22 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 16:22 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Add python3-dev[el] to bindep.txt https://review.opendev.org/709694 | 16:23 |
| cosmicsound | yoctozepto , what was the command to get to the one file .pem | 16:35 |
| cosmicsound | thats the normal crt with the .key into one .pem? | 16:36 |
| yoctozepto | yeah, they are concatenated | 16:36 |
| cosmicsound | -----END CERTIFICATE----- | 16:36 |
| cosmicsound | -----BEGIN RSA PRIVATE KEY----- | 16:36 |
| cosmicsound | i have first cert and then followed by key | 16:36 |
| cosmicsound | inside nothing elese | 16:36 |
| cosmicsound | il give it a go | 16:36 |
| cosmicsound | so no eed for ca_chain] | 16:36 |
| yoctozepto | I have cert, cert and key | 16:37 |
| yoctozepto | whole chain to allow for match with root | 16:37 |
| cosmicsound | not sure i get this | 16:38 |
| cosmicsound | do you take your instance public? | 16:39 |
| cosmicsound | or is just internal tls | 16:39 |
| cosmicsound | root = ca authority who made the ssl? in my case is sectigo old comodo | 16:39 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 16:42 |
| yoctozepto | cosmicsound: certs are usually signed by an intermediate | 16:43 |
| yoctozepto | cosmicsound: and only root is trusted | 16:43 |
| yoctozepto | cosmicsound: so omitting intermediary from cert chain may render the connection untrusted | 16:43 |
| hrw | mgoddard: found the reason for mistral-dashboard going backwards... | 16:43 |
| hrw | mgoddard: will discuss with release team | 16:44 |
| cosmicsound | yes so i need the provider trust chain | 16:44 |
| cosmicsound | got it | 16:44 |
| *** bengates has quit IRC | 16:50 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: Bump train versions https://review.opendev.org/710067 | 16:53 |
| cosmicsound | redeploying 🤞 | 17:05 |
| cosmicsound | got it into one .pem all | 17:05 |
| cosmicsound | ca chain cert and key | 17:05 |
| cosmicsound | i am confused a bit with node_custom_config and node_config are they same? | 17:06 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Move ansible installation & configuration to Ansible https://review.opendev.org/709717 | 17:06 |
| cosmicsound | node_custom_config = /etc/kolla/config | 17:06 |
| cosmicsound | certs have the {{ode_custom}}/certificates . do i need to put them in /config ? | 17:07 |
| cosmicsound | *node | 17:07 |
| ktibi | mgoddard, when bifrost start I can see in the container a process: git-remote-https origin https://opendev.org/openstack/ironic | 17:10 |
| ktibi | the image need to be have internet access ? | 17:10 |
| mgoddard | ktibi: unfortunately, yes | 17:14 |
| ktibi | mgoddard, hum, any workarround ? | 17:15 |
| mgoddard | ktibi: maybe you can pass some --skip-tags to bifrost? | 17:15 |
| mgoddard | or add config to point to local repos | 17:15 |
| ktibi | the playbook install bifrost in the container during the bootstrap ? | 17:16 |
| mgoddard | ktibi: we run bifrost install when the container is created | 17:17 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Use upper constraints when installing clients https://review.opendev.org/709751 | 17:17 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: CI: Use python 3 for local kolla-ansible execution https://review.opendev.org/709718 | 17:17 |
| mgoddard | then again with some --skip-tags during bootstrap | 17:17 |
| mgoddard | maybe we are missing some tags to skip? | 17:17 |
| ktibi | when you say "is created" it's during the build of the image ? | 17:18 |
| mgoddard | yes | 17:19 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI https://review.opendev.org/709536 | 17:19 |
| ktibi | mgoddard, ok I can see a task in bifrost-prep, Download via GIT with a var bifrost_install_sources | 17:21 |
| ktibi | how with kayobe can I override variables in this role ? | 17:21 |
| cosmicsound | +1 yoctozepto | 17:21 |
| cosmicsound | loving kolla more and more each time i pass one messy one like this | 17:21 |
| cosmicsound | https://mdb.uhlhost.net/uploads/b09146c0dbe1a931/image.png | 17:21 |
| cosmicsound | TLS works on public domain | 17:22 |
| mgoddard | ktibi: https://docs.openstack.org/kayobe/latest/configuration/bifrost.html#custom-configuration | 17:22 |
| cosmicsound | now need to solve internal domain naming | 17:22 |
| ktibi | mgoddard, but I don't understand why I need to clone ironic repo. because it's a source image ? and not a binary ? | 17:22 |
| hrw | bye | 17:23 |
| mgoddard | ktibi: it's just part of bifrost installation | 17:24 |
| ktibi | yes but the installation is during the build, not during the run no ? | 17:25 |
| ktibi | like binary image | 17:25 |
| ktibi | mgoddard, because I can see the call to the install playbook in the dockerfile, but why kayobe run again this playbook during the bootstrap of image :/ | 17:30 |
| ktibi | mgoddard, ok I found, in kolla-ansible it's skip_package_install=true but in the playbook it's when: skip_install is not defined | 17:34 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-kolla | 17:35 | |
| ktibi | I try to add skip_install: true with your solution :) works | 17:40 |
| *** gfidente is now known as gfidente|afk | 17:40 | |
| * yoctozepto is glad, cosmicsound | 17:44 | |
| * cosmicsound is gratefull for such a community | 17:45 | |
| cosmicsound | The only one think I did not managed to get going | 17:46 |
| cosmicsound | was allocation of ipv4 vip external | 17:46 |
| cosmicsound | while if you remember on prechecks it used to pass the pinng | 17:46 |
| cosmicsound | in the end no horizon was accessible | 17:46 |
| cosmicsound | the bug i opened tried to explain this | 17:46 |
| yoctozepto | cosmicsound: where are you showing the cert being deployed then? ;p | 17:47 |
| cosmicsound | well this is made with haproxy off and keepalive | 17:47 |
| *** ktibi has quit IRC | 17:48 | |
| cosmicsound | and i used main public ip assigned to my bare servers | 17:48 |
| cosmicsound | I have eno1 public and eno2 internal | 17:48 |
| cosmicsound | 172.22.0.1/24 with .22 internal vip and eno1 main ipv4 as external ip yet not VIP since haproxy is off | 17:49 |
| *** k_mouza has quit IRC | 17:49 | |
| cosmicsound | with haproxy on it wont work, i added another address set it was same no ping so il debug this more | 17:50 |
| cosmicsound | at first its netplan causing it | 17:50 |
| cosmicsound | will try also bond approach since now its flat | 17:50 |
| cosmicsound | bridge sorry | 17:50 |
| cosmicsound | Now i made the main ssl work, will give it another chance | 17:51 |
| cosmicsound | need more rtm on neutron | 17:52 |
| cosmicsound | i know il make a terminal record session | 17:53 |
| cosmicsound | is easiest to see what is done | 17:53 |
| *** dougsz has quit IRC | 17:57 | |
| *** diurnalist has quit IRC | 18:07 | |
| *** lile has quit IRC | 18:11 | |
| *** tonythomas has quit IRC | 18:21 | |
| *** diurnalist has joined #openstack-kolla | 18:25 | |
| *** lile has joined #openstack-kolla | 18:27 | |
| *** priteau has quit IRC | 18:32 | |
| *** igordc has joined #openstack-kolla | 18:37 | |
| *** igordc has quit IRC | 18:43 | |
| openstackgerrit | Merged openstack/kayobe master: CentOS 8: Use ansible_playbook_python for localhost dependencies https://review.opendev.org/708921 | 18:45 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 18:48 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Fix fernet bootstrap and key distribution - follow up https://review.opendev.org/707080 | 18:49 |
| osmanlicilegi | need a quick review for https://review.opendev.org/#/c/705608/ | 18:49 |
| patchbot | patch 705608 - kolla-ansible - Fixes gnocchi-api script name for Ubuntu/Debian - 1 patch set | 18:49 |
| mnasiadka | osmanlicilegi: done | 18:51 |
| osmanlicilegi | thanks! | 18:51 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Add /run/netns bindmount to Neutron containers https://review.opendev.org/710051 | 18:54 |
| cosmicsound | yoctozepto , this one is the answer on my precheks now | 19:39 |
| cosmicsound | TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] *********************************************** | 19:39 |
| cosmicsound | ok: [compute-1] => (item={'address': '172.22.0.22', 'command': 'ping'}) | 19:39 |
| cosmicsound | failed: [compute-1] (item={'address': '51.91.153.141', 'command': 'ping'}) => {"ansible_loop_var": "item", "changed": false, "cmd": ["ping", "-c", "3", "51.91.153.141"], "delta": "0:00:02.050088", "end": "2020-02-26 19:38:08.288743", "failed_when_result": true, "item": {"address": "51.91.153.141", "command": "ping"}, "rc": 0, "start": "2020-02-26 | 19:39 |
| cosmicsound | 19:38:06.238655", "stderr": "", "stderr_lines": [], "stdout": "PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.\n64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms\n64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms\n64 bytes from 51.91.153.141: icmp_seq=3 ttl=64 time=0.084 ms\n\n--- 51.91.153.141 ping statistics ---\n3 | 19:39 |
| cosmicsound | packets transmitted, 3 received, 0% packet loss, time 2048ms\nrtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms", "stdout_lines": ["PING 51.91.153.141 (51.91.153.141) 56(84) bytes of data.", "64 bytes from 51.91.153.141: icmp_seq=1 ttl=64 time=0.022 ms", "64 bytes from 51.91.153.141: icmp_seq=2 ttl=64 time=0.077 ms", "64 bytes from 51.91.153.141: | 19:39 |
| cosmicsound | icmp_seq=3 ttl=64 time=0.084 ms", "", "--- 51.91.153.141 ping statistics ---", "3 packets transmitted, 3 received, 0% packet loss, time 2048ms", "rtt min/avg/max/mdev = 0.022/0.061/0.084/0.027 ms"]} | 19:39 |
| cosmicsound | alto the ip its pingable | 19:39 |
| cosmicsound | its indeed not a floating ip, with a flowting ip this will pass just my dashboard will be gone | 19:39 |
| yoctozepto | it must *not* be pingable *before* deployment | 19:39 |
| yoctozepto | because it will be configured | 19:40 |
| cosmicsound | right | 19:40 |
| cosmicsound | because indeed this replies to pings | 19:40 |
| cosmicsound | last log i showed you it passed just no console anywhere | 19:41 |
| cosmicsound | will retry that range | 19:41 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible stable/train: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710126 | 19:55 |
| openstackgerrit | Dincer Celik proposed openstack/kolla-ansible stable/stein: Use more permissive regex to remove the offending 127.0.1.1 https://review.opendev.org/710127 | 19:55 |
| yoctozepto | osmanlicilegi: no rocky? ^ | 20:02 |
| *** diurnalist has quit IRC | 20:26 | |
| *** kozhukalov has quit IRC | 20:33 | |
| *** kozhukalov has joined #openstack-kolla | 20:36 | |
| *** sean-k-mooney has quit IRC | 20:41 | |
| *** diurnalist has joined #openstack-kolla | 20:41 | |
| *** kplant has quit IRC | 20:49 | |
| openstackgerrit | Merged openstack/kolla stable/stein: Ignore EM releases in version-check.py https://review.opendev.org/710073 | 21:09 |
| openstackgerrit | Merged openstack/kolla stable/rocky: Ignore EM releases in version-check.py https://review.opendev.org/710074 | 21:09 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fixes gnocchi-api script name for Ubuntu/Debian https://review.opendev.org/705608 | 21:09 |
| *** priteau has joined #openstack-kolla | 21:28 | |
| *** kozhukalov has quit IRC | 21:29 | |
| *** k_mouza has joined #openstack-kolla | 21:50 | |
| *** k_mouza has quit IRC | 21:55 | |
| *** priteau has quit IRC | 22:11 | |
| *** JangwonLee has quit IRC | 22:31 | |
| *** TrevorV has quit IRC | 22:38 | |
| *** ktibi has joined #openstack-kolla | 23:04 | |
| *** ktibi has quit IRC | 23:09 | |
| *** JangwonLee has joined #openstack-kolla | 23:11 | |
| *** cah_link1 has joined #openstack-kolla | 23:20 | |
| *** cah_link has quit IRC | 23:23 | |
| *** cah_link1 is now known as cah_link | 23:23 | |
| *** JangwonLee has quit IRC | 23:45 | |
| *** JangwonLee has joined #openstack-kolla | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
