15:00:19 #startmeeting kolla 15:00:19 Meeting started Wed Mar 16 15:00:19 2022 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:19 The meeting name has been set to 'kolla' 15:00:25 #topic rollcall 15:00:26 o/ 15:00:29 \o 15:00:47 o\ 15:01:04 o/ 15:03:01 \o 15:03:36 #topic agenda 15:03:36 * Announcements 15:03:36 * Review action items from the last meeting 15:03:36 * CI status 15:03:36 * Release tasks 15:03:37 * Current cycle planning 15:03:37 * Additional agenda (from whiteboard) 15:03:39 * Open discussion 15:03:41 #topic Announcements 15:04:06 I've proposed frickler to gain core reviewer privileges in both Kolla and Kolla-Ansible - please vote on the mailing list. 15:04:16 #topic Review action items from the last meeting 15:04:32 mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:04:32 mnasiadka to update kolla review dashboard with kolla collection 15:04:32 hrw Enable osbpo in Debian APT sources, abandon extrepo command use then? 15:04:39 I updated the review dashboard 15:05:20 sec bugs in progress, I proposed a patch to use testssl.sh to validate haproxy ssl ciphers (one of the bugs mentions ssl ciphers) 15:05:42 needs some update, but should ensure this is covered a bit better than in past 15:05:58 hrw is not here, so let's reapply this action item 15:06:21 #action mnasiadka to triage security bugs and update them with resolution plan (if needed) 15:06:28 #action hrw Enable osbpo in Debian APT sources, abandon extrepo command use then? 15:06:33 #topic CI status 15:06:48 We've seen breakage from ansible-lint complaining on us not using FQCNs for core modules 15:07:03 it has been capped to <6 - it's fine now 15:07:07 ubuntu-binary seems also broken 15:07:16 some issue with trove-dashboard afaict 15:07:26 https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_de8/793897/33/check/kolla-ansible-ubuntu-binary/de8b1e3/primary/logs/kolla/horizon/horizon.txt 15:07:29 that's why we are dropping support for binary 15:07:36 well, one of the reasons 15:08:02 but as of now we still should try to fix it? 15:08:05 I think i've seen something similar in the Magnum CI job (where we enable Trove I think) 15:08:23 frickler: at least we should raise a Launchpad bug in UCA 15:08:28 that's what we've done in the past 15:09:11 yeah, raise a bug in UCA 15:09:15 ah, o.k., I can do that and ping some ubuntu-server ppl 15:09:49 but they're usually not the fastest, so it will drag around 15:09:59 and IIRC Trove doesn't have a PTL for Z 15:10:12 yes 15:10:22 So I don't know how long will it stay with us ;) 15:10:27 anyway, let's continue 15:10:37 #topic Release tasks 15:11:25 wuchunyang stepped up for trove PTL 15:11:46 oh, good to know, I missed that 15:12:05 so it's only adjutant going away 15:12:20 anyway it would be good to understand if we don't hit that in source as well - I've seen trove being enabled in magnum jobs 15:12:23 thankfully, we did not merge its support 15:12:45 Kolla feature freeze: Mar 21 - Mar 25 15:12:47 yes, we will maintain trove in z 15:12:55 It's next week 15:13:11 #topic Current cycle planning 15:13:26 So, what is possibly going to get merged? 15:13:39 mnasiadka: we always have all horizon plugins around so it would break source already, no? 15:13:56 mnasiadka: I've reviewed venus 15:14:06 i've also added it to the other topics for today 15:14:07 venus +2 from me 15:14:10 feel free to skip then 15:14:16 yoctozepto: True, we only enable plugins per project I think, but who knows what fails 15:14:19 ok 15:14:22 so - what about - Systemd containers: https://review.opendev.org/c/openstack/kolla-ansible/+/816724 15:14:36 mgoddard, yoctozepto: you've been actively reviewing, right? 15:14:56 I haven't looked recently 15:15:16 me neither 15:15:23 Ok, would be nice to get this in though, right? 15:15:49 right 15:16:01 and my ironic patches 15:16:04 :-) 15:16:35 we'll get to that (if those are listed on the whiteboard) :) 15:17:11 kolla collection long list of patches - yoctozepto you promised to review two, haven't done that :D 15:17:23 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/821015 15:17:23 https://review.opendev.org/c/openstack/ansible-collection-kolla/+/821016 15:17:36 and there are like 5 more on the list 15:17:45 L433 on the whiteboard 15:17:49 mnasiadka: oh noez, so many things to do 15:18:09 need reviewers, if not - it's not going to get in 15:18:25 maybe there are some other cores happy to assist? 15:19:04 anyway, let's move on - libvirt on host the same - I'll do reviews and need a second core 15:19:13 Will Szumski proposed openstack/kolla-ansible master: Adds keystone_authtoken.service_type https://review.opendev.org/c/openstack/kolla-ansible/+/834035 15:19:21 and magically Ironic appeared on the whiteboard! 15:19:24 I think kevko approved libvirt on the host 15:19:43 but not the kolla collection patch 15:19:46 mnasiadka: yup, it's magix 15:20:14 true 15:21:20 ok, let's review Radek's Ironic patches, and he'll be happy to review all the rest! ;-) 15:22:19 Kayobe seems we have a lot of patches that need updates 15:22:40 probably the multiple environments part 2 won't be merged in Yoga 15:22:49 :( 15:23:15 Haven't seen updates on those patches, and there's a lot to improve judging by mgoddard's comments 15:23:50 Will was planning to update but he's been busy 15:24:00 As we all are unfortunately 15:24:15 Let's go through additional agenda 15:24:18 #topic Additional agenda (from whiteboard) 15:24:35 (yoctozepto) Venus 15:25:01 So what's up with Venus? 15:26:57 mnasiadka: already discussed, move on 15:27:00 ok 15:27:08 If we're at Venus - what's up with skyline? 15:27:30 I did not have time / it was less active/ready? 15:27:32 let's check 15:27:40 i can try to add a ci for skyline. 15:27:59 follow by venus. 15:28:09 wuchunyang: that would be appreciated 15:28:44 https://review.opendev.org/c/openstack/kolla-ansible/+/828464 15:29:08 (fwiw, I'm reviewing various changes atm) 15:29:54 wuchunyang: the Kolla part has some comments, like why it's not using upper-constraints, I see Skyline would need more work than Venus to get in 15:30:56 Pierre Riteau proposed openstack/kayobe master: Use naming convention to infer VLAN tagging https://review.opendev.org/c/openstack/kayobe/+/833052 15:30:59 Ok, let's keep reviewing - hope the authors will update 15:31:02 yes, skyline need more work. 15:31:22 Next additional topic is (mgoddard): Libvirt SASL issues 15:31:50 I had some issues today when rolling out the libvirt SASL change 15:32:10 I'm still not really sure exactly what happened, but wanted to bring it up 15:32:52 Possibly there is a window when enabling SASL where it breaks the connection from nova-compute to libvirt 15:33:03 and some instances get broken 15:33:09 oops 15:33:30 oh my 15:33:43 how broken are we talking about? 15:33:45 data loss? 15:33:59 or just powered off / wrong state in Nova? 15:34:39 libvirt VM stopped 15:35:01 I managed to get it running again with an openstack server stop, openstack server start 15:35:18 hmm 15:35:24 but it took me a while to figure it out 15:35:30 I don't see how it would stop the instance by itself 15:35:39 maybe it was on user req? 15:35:44 but the sync back did not happen 15:35:56 I'm not entirely sure when nova compute updates its state 15:36:34 when backporting the libvirt sasl patch - did we enable sasl by default? 15:36:38 yes 15:36:54 so basically this can now happen to anyone? 15:37:06 yes 15:37:10 potentially, but haven't reproduced it 15:37:32 interesting 15:38:02 if my theory is correct, we could stop nova-compute first, then restart nova-libvirt, then start nova-compute 15:38:16 that would be a simple fix to backport 15:38:55 but really I'd like to reproduce it to be confident 15:39:25 ++ 15:39:37 Sure, we'll be waiting for updates mgoddard 15:40:51 ok 15:42:15 #topic Open discussion 15:42:21 Hello guys. Wanna ask your advice about separating LB for several groups. My colleague tried to do some [https://review.opendev.org/c/openstack/kolla-ansible/+/833535]. But for now we can't decide which way to choose, simple one (just a small patch to separate ELK's LB (what we want to achieve for now)) or hard one (provide some way to have an option for several LB groups with service groups mapping to them). Thanks in advance for 15:43:40 mgoddard: seems you've been giving your thoughts on this one ^^ 15:44:23 yes 15:44:57 the generic solution would be to have some flag per service 15:45:14 yup sounds reasonable 15:45:15 elasticsearch_enable_loadbalancer 15:45:42 then incorporate that into elasticsearch_services.elasticsearch.haproxy 15:45:52 but loadbalancer might not be on the same nodes as these services 15:45:57 and it's fine 15:46:08 it's even on network nodes, not control nodes 15:46:21 yes, that was my comment 15:46:25 what about hostgroup_vip 15:46:35 but in that case VIP will be attached to host_group? 15:46:55 the VIP is associated with haproxy hosts 15:47:05 yep 15:47:08 i.e., we have one loadbalancer solution 15:47:16 that is HA 15:47:24 and shuffles VIP to ensure that HA 15:47:39 to separete newtwork flow 15:47:46 if you have your own LB for elastic, you'd need to set elasticsearch_enable_loadbalancer=false and elasticsearch_internal_endpoint to point to your LB 15:47:49 the proxied services might exists anywhere 15:48:20 I have monitoring_vip and a several services on this group 15:48:46 or perhaps you could just set elasticsearch_address 15:48:54 I did 15:49:40 so the way where there might be more than 1 LB in HA placed on control nodes isn't the way you want to see? :] 15:49:40 why we cant have itsown loadbalancer on group 15:50:36 oh, so you want kolla to deploy multiple LBs? 15:50:40 I had Virtual contrail Plane at least 9 nodes with 15:50:42 yes 15:50:49 I have it already 15:50:54 to have that possibility 15:51:21 what do you use multiple LBs for? only es? 15:51:43 we have the same scenario. i use kolla to deploy two lbs, but es exists in both them. 15:51:45 as first run 15:51:46 by default okay. everything is going thorugh control-nodes. by for some cases it might be unnecessary and should be avoidable. like ELK-case 15:52:05 choooze: it goes through network nodes 15:52:15 you can have a separate network node from control nodes 15:52:25 (or multiple for that matter) 15:52:38 anyway I hope you've got the point :] 15:54:12 so-so I'd say 15:54:44 to me it makes sense to have a separate loadbalancer for elk 15:54:52 well, more like efk 15:54:56 there are core components of Openstack and additional services - And I want' mix traffic 15:55:57 But that means, you'd want to have a second instance of keepalived and haproxy - and possibly on a different set of physical hosts? 15:56:11 yup 15:56:51 2 strings into haproxyconfig.j2 :0 15:56:56 Well, sounds like a nice feature, that we could discuss over PTG for the Z cycle. 15:57:43 ++ 15:58:05 Nice guys! 15:58:16 sounds similar to the multiple mariadb feature, which was quite fiddly to get right 15:58:38 We could help somehow if you'll show the right way :] 15:59:10 Sure, please add a topic on the bottom of Zed etherpad - https://etherpad.opendev.org/p/kolla-zed-ptg 15:59:39 and of course show up for the discussion :) 15:59:48 Update on the libvirt SASL issue discussed earlier: I think this was actually caused by some previous failed evacuations. nova-compute restart caused them to get cleaned up, which destroyed the instances 16:00:22 Imran Hussain proposed openstack/kolla-ansible master: [external-ceph] Use template instead of copy https://review.opendev.org/c/openstack/kolla-ansible/+/824566 16:00:44 mnasiadka next Wed you mean? or some other time? 16:00:46 mgoddard: should we make an update in the reno - that this might cause unplanned instance downtime? 16:01:03 mnasiadka: no, it was unrelated to libvirt SASL AFAICT 16:01:24 I will keep investigating 16:01:36 choooze: PTG (Project Teams Gathering) is 4-6 April (Kolla-Ansible topics 4-5 April) 16:01:37 thanks 16:01:58 okaaaay sounds nice 16:02:06 thank you guys! 16:03:01 choooze: I added an Eventbrite link for the event, make sure you order a virtual ticket ;-) 16:03:26 ok, we're over time. 16:03:28 Thanks for coming! 16:03:30 #endmeeting