15:00:19 <mnasiadka> #startmeeting kolla
15:00:19 <opendevmeet> Meeting started Wed Mar 16 15:00:19 2022 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:19 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:19 <opendevmeet> The meeting name has been set to 'kolla'
15:00:25 <mnasiadka> #topic rollcall
15:00:26 <mnasiadka> o/
15:00:29 <frickler> \o
15:00:47 <jinyuanliu_> o\
15:01:04 <yoctozepto> o/
15:03:01 <mgoddard> \o
15:03:36 <mnasiadka> #topic agenda
15:03:36 <mnasiadka> * Announcements
15:03:36 <mnasiadka> * Review action items from the last meeting
15:03:36 <mnasiadka> * CI status
15:03:36 <mnasiadka> * Release tasks
15:03:37 <mnasiadka> * Current cycle planning
15:03:37 <mnasiadka> * Additional agenda (from whiteboard)
15:03:39 <mnasiadka> * Open discussion
15:03:41 <mnasiadka> #topic Announcements
15:04:06 <mnasiadka> I've proposed frickler to gain core reviewer privileges in both Kolla and Kolla-Ansible - please vote on the mailing list.
15:04:16 <mnasiadka> #topic Review action items from the last meeting
15:04:32 <mnasiadka> mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:04:32 <mnasiadka> mnasiadka to update kolla review dashboard with kolla collection
15:04:32 <mnasiadka> hrw Enable osbpo in Debian APT sources, abandon extrepo command use then?
15:04:39 <mnasiadka> I updated the review dashboard
15:05:20 <mnasiadka> sec bugs in progress, I proposed a patch to use testssl.sh to validate haproxy ssl ciphers (one of the bugs mentions ssl ciphers)
15:05:42 <mnasiadka> needs some update, but should ensure this is covered a bit better than in past
15:05:58 <mnasiadka> hrw is not here, so let's reapply this action item
15:06:21 <mnasiadka> #action mnasiadka to triage security bugs and update them with resolution plan (if needed)
15:06:28 <mnasiadka> #action hrw Enable osbpo in Debian APT sources, abandon extrepo command use then?
15:06:33 <mnasiadka> #topic CI status
15:06:48 <mnasiadka> We've seen breakage from ansible-lint complaining on us not using FQCNs for core modules
15:07:03 <mnasiadka> it has been capped to <6 - it's fine now
15:07:07 <frickler> ubuntu-binary seems also broken
15:07:16 <frickler> some issue with trove-dashboard afaict
15:07:26 <frickler> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_de8/793897/33/check/kolla-ansible-ubuntu-binary/de8b1e3/primary/logs/kolla/horizon/horizon.txt
15:07:29 <yoctozepto> that's why we are dropping support for binary
15:07:36 <yoctozepto> well, one of the reasons
15:08:02 <frickler> but as of now we still should try to fix it?
15:08:05 <mnasiadka> I think i've seen something similar in the Magnum CI job (where we enable Trove I think)
15:08:23 <mnasiadka> frickler: at least we should raise a Launchpad bug in UCA
15:08:28 <mnasiadka> that's what we've done in the past
15:09:11 <yoctozepto> yeah, raise a bug in UCA
15:09:15 <frickler> ah, o.k., I can do that and ping some ubuntu-server ppl
15:09:49 <mnasiadka> but they're usually not the fastest, so it will drag around
15:09:59 <mnasiadka> and IIRC Trove doesn't have a PTL for Z
15:10:12 <yoctozepto> yes
15:10:22 <mnasiadka> So I don't know how long will it stay with us ;)
15:10:27 <mnasiadka> anyway, let's continue
15:10:37 <mnasiadka> #topic Release tasks
15:11:25 <mgoddard> wuchunyang stepped up for trove PTL
15:11:46 <yoctozepto> oh, good to know, I missed that
15:12:05 <yoctozepto> so it's only adjutant going away
15:12:20 <mnasiadka> anyway it would be good to understand if we don't hit that in source as well - I've seen trove being enabled in magnum jobs
15:12:23 <yoctozepto> thankfully, we did not merge its support
15:12:45 <mnasiadka> Kolla feature freeze: Mar 21 - Mar 25
15:12:47 <wuchunyang> yes, we will maintain trove in z
15:12:55 <mnasiadka> It's next week
15:13:11 <mnasiadka> #topic Current cycle planning
15:13:26 <mnasiadka> So, what is possibly going to get merged?
15:13:39 <yoctozepto> mnasiadka: we always have all horizon plugins around so it would break source already, no?
15:13:56 <yoctozepto> mnasiadka: I've reviewed venus
15:14:06 <yoctozepto> i've also added it to the other topics for today
15:14:07 <wuchunyang> venus +2 from me
15:14:10 <yoctozepto> feel free to skip then
15:14:16 <mnasiadka> yoctozepto: True, we only enable plugins per project I think, but who knows what fails
15:14:19 <mnasiadka> ok
15:14:22 <mnasiadka> so - what about - Systemd containers: https://review.opendev.org/c/openstack/kolla-ansible/+/816724
15:14:36 <mnasiadka> mgoddard, yoctozepto: you've been actively reviewing, right?
15:14:56 <mgoddard> I haven't looked recently
15:15:16 <yoctozepto> me neither
15:15:23 <mnasiadka> Ok, would be nice to get this in though, right?
15:15:49 <yoctozepto> right
15:16:01 <yoctozepto> and my ironic patches
15:16:04 <yoctozepto> :-)
15:16:35 <mnasiadka> we'll get to that (if those are listed on the whiteboard) :)
15:17:11 <mnasiadka> kolla collection long list of patches - yoctozepto you promised to review two, haven't done that :D
15:17:23 <mnasiadka> https://review.opendev.org/c/openstack/ansible-collection-kolla/+/821015
15:17:23 <mnasiadka> https://review.opendev.org/c/openstack/ansible-collection-kolla/+/821016
15:17:36 <mnasiadka> and there are like 5 more on the list
15:17:45 <mnasiadka> L433 on the whiteboard
15:17:49 <yoctozepto> mnasiadka: oh noez, so many things to do
15:18:09 <mnasiadka> need reviewers, if not - it's not going to get in
15:18:25 <mnasiadka> maybe there are some other cores happy to assist?
15:19:04 <mnasiadka> anyway, let's move on - libvirt on host the same - I'll do reviews and need a second core
15:19:13 <opendevreview> Will Szumski proposed openstack/kolla-ansible master: Adds keystone_authtoken.service_type  https://review.opendev.org/c/openstack/kolla-ansible/+/834035
15:19:21 <mnasiadka> and magically Ironic appeared on the whiteboard!
15:19:24 <mgoddard> I think kevko approved libvirt on the host
15:19:43 <mnasiadka> but not the kolla collection patch
15:19:46 <yoctozepto> mnasiadka: yup, it's magix
15:20:14 <mgoddard> true
15:21:20 <mnasiadka> ok, let's review Radek's Ironic patches, and he'll be happy to review all the rest! ;-)
15:22:19 <mnasiadka> Kayobe seems we have a lot of patches that need updates
15:22:40 <mnasiadka> probably the multiple environments part 2 won't be merged in Yoga
15:22:49 <priteau> :(
15:23:15 <mnasiadka> Haven't seen updates on those patches, and there's a lot to improve judging by mgoddard's comments
15:23:50 <priteau> Will was planning to update but he's been busy
15:24:00 <mnasiadka> As we all are unfortunately
15:24:15 <mnasiadka> Let's go through additional agenda
15:24:18 <mnasiadka> #topic Additional agenda (from whiteboard)
15:24:35 <mnasiadka> (yoctozepto) Venus
15:25:01 <mnasiadka> So what's up with Venus?
15:26:57 <yoctozepto> mnasiadka: already discussed, move on
15:27:00 <mnasiadka> ok
15:27:08 <mnasiadka> If we're at Venus - what's up with skyline?
15:27:30 <yoctozepto> I did not have time / it was less active/ready?
15:27:32 <yoctozepto> let's check
15:27:40 <wuchunyang> i can try to add a ci for skyline.
15:27:59 <wuchunyang> follow by venus.
15:28:09 <yoctozepto> wuchunyang: that would be appreciated
15:28:44 <wuchunyang> https://review.opendev.org/c/openstack/kolla-ansible/+/828464
15:29:08 <yoctozepto> (fwiw, I'm reviewing various changes atm)
15:29:54 <mnasiadka> wuchunyang: the Kolla part has some comments, like why it's not using upper-constraints, I see Skyline would need more work than Venus to get in
15:30:56 <opendevreview> Pierre Riteau proposed openstack/kayobe master: Use naming convention to infer VLAN tagging  https://review.opendev.org/c/openstack/kayobe/+/833052
15:30:59 <mnasiadka> Ok, let's keep reviewing - hope the authors will update
15:31:02 <wuchunyang> yes, skyline need more work.
15:31:22 <mnasiadka> Next additional topic is (mgoddard): Libvirt SASL issues
15:31:50 <mgoddard> I had some issues today when rolling out the libvirt SASL change
15:32:10 <mgoddard> I'm still not really sure exactly what happened, but wanted to bring it up
15:32:52 <mgoddard> Possibly there is a window when enabling SASL where it breaks the connection from nova-compute to libvirt
15:33:03 <mgoddard> and some instances get broken
15:33:09 <mnasiadka> oops
15:33:30 <yoctozepto> oh my
15:33:43 <yoctozepto> how broken are we talking about?
15:33:45 <yoctozepto> data loss?
15:33:59 <mnasiadka> or just powered off / wrong state in Nova?
15:34:39 <mgoddard> libvirt VM stopped
15:35:01 <mgoddard> I managed to get it running again with an openstack server stop, openstack server start
15:35:18 <yoctozepto> hmm
15:35:24 <mgoddard> but it took me a while to figure it out
15:35:30 <yoctozepto> I don't see how it would stop the instance by itself
15:35:39 <yoctozepto> maybe it was on user req?
15:35:44 <yoctozepto> but the sync back did not happen
15:35:56 <yoctozepto> I'm not entirely sure when nova compute updates its state
15:36:34 <mnasiadka> when backporting the libvirt sasl patch - did we enable sasl by default?
15:36:38 <mgoddard> yes
15:36:54 <mnasiadka> so basically this can now happen to anyone?
15:37:06 <yoctozepto> yes
15:37:10 <mgoddard> potentially, but haven't reproduced it
15:37:32 <mnasiadka> interesting
15:38:02 <mgoddard> if my theory is correct, we could stop nova-compute first, then restart nova-libvirt, then start nova-compute
15:38:16 <mgoddard> that would be a simple fix to backport
15:38:55 <mgoddard> but really I'd like to reproduce it to be confident
15:39:25 <yoctozepto> ++
15:39:37 <mnasiadka> Sure, we'll be waiting for updates mgoddard
15:40:51 <mgoddard> ok
15:42:15 <mnasiadka> #topic Open discussion
15:42:21 <choooze> Hello guys. Wanna ask your advice about separating LB for several groups. My colleague tried to do some [https://review.opendev.org/c/openstack/kolla-ansible/+/833535]. But for now we can't decide which way to choose, simple one (just a small patch to separate ELK's LB (what we want to achieve for now)) or hard one (provide some way to have an option for several LB groups with service groups mapping to them). Thanks in advance for
15:43:40 <mnasiadka> mgoddard: seems you've been giving your thoughts on this one ^^
15:44:23 <mgoddard> yes
15:44:57 <mgoddard> the generic solution would be to have some flag per service
15:45:14 <choooze> yup sounds reasonable
15:45:15 <mgoddard> elasticsearch_enable_loadbalancer
15:45:42 <mgoddard> then incorporate that into elasticsearch_services.elasticsearch.haproxy
15:45:52 <yoctozepto> but loadbalancer might not be on the same nodes as these services
15:45:57 <yoctozepto> and it's fine
15:46:08 <yoctozepto> it's even on network nodes, not control nodes
15:46:21 <mgoddard> yes, that was my comment
15:46:25 <jingvar> what about hostgroup_vip
15:46:35 <choooze> but in that case VIP will be attached to host_group?
15:46:55 <mgoddard> the VIP is associated with haproxy hosts
15:47:05 <jingvar> yep
15:47:08 <yoctozepto> i.e., we have one loadbalancer solution
15:47:16 <yoctozepto> that is HA
15:47:24 <yoctozepto> and shuffles VIP to ensure that HA
15:47:39 <jingvar> to separete newtwork flow
15:47:46 <mgoddard> if you have your own LB for elastic, you'd need to set elasticsearch_enable_loadbalancer=false and elasticsearch_internal_endpoint to point to your LB
15:47:49 <yoctozepto> the proxied services might exists anywhere
15:48:20 <jingvar> I have monitoring_vip and a several services on this group
15:48:46 <mgoddard> or perhaps you could just set elasticsearch_address
15:48:54 <jingvar> I did
15:49:40 <choooze> so the way where there might be more than 1 LB in HA placed on control nodes isn't the way you want to see? :]
15:49:40 <jingvar> why we cant have itsown loadbalancer on group
15:50:36 <mgoddard> oh, so you want kolla to deploy multiple LBs?
15:50:40 <jingvar> I had Virtual contrail Plane at least 9 nodes with
15:50:42 <jingvar> yes
15:50:49 <jingvar> I have it already
15:50:54 <choooze> to have that possibility
15:51:21 <yoctozepto> what do you use multiple LBs for? only es?
15:51:43 <wuchunyang> we have the same scenario. i use kolla to deploy two lbs, but es exists in both them.
15:51:45 <jingvar> as first run
15:51:46 <choooze> by default okay. everything is going thorugh control-nodes. by for some cases it might be unnecessary and should be avoidable. like ELK-case
15:52:05 <yoctozepto> choooze: it goes through network nodes
15:52:15 <yoctozepto> you can have a separate network node from control nodes
15:52:25 <yoctozepto> (or multiple for that matter)
15:52:38 <choooze> anyway I hope you've got the point :]
15:54:12 <yoctozepto> so-so I'd say
15:54:44 <yoctozepto> to me it makes sense to have a separate loadbalancer for elk
15:54:52 <yoctozepto> well, more like efk
15:54:56 <jingvar> there are core components of Openstack and additional services - And I want' mix traffic
15:55:57 <mnasiadka> But that means, you'd want to have a second instance of keepalived and haproxy - and possibly on a different set of physical hosts?
15:56:11 <choooze> yup
15:56:51 <jingvar> 2 strings into haproxyconfig.j2 :0
15:56:56 <mnasiadka> Well, sounds like a nice feature, that we could discuss over PTG for the Z cycle.
15:57:43 <yoctozepto> ++
15:58:05 <choooze> Nice guys!
15:58:16 <mgoddard> sounds similar to the multiple mariadb feature, which was quite fiddly to get right
15:58:38 <choooze> We could help somehow if you'll show the right way :]
15:59:10 <mnasiadka> Sure, please add a topic on the bottom of Zed etherpad - https://etherpad.opendev.org/p/kolla-zed-ptg
15:59:39 <mnasiadka> and of course show up for the discussion :)
15:59:48 <mgoddard> Update on the libvirt SASL issue discussed earlier: I think this was actually caused by some previous failed evacuations. nova-compute restart caused them to get cleaned up, which destroyed the instances
16:00:22 <opendevreview> Imran Hussain proposed openstack/kolla-ansible master: [external-ceph] Use template instead of copy  https://review.opendev.org/c/openstack/kolla-ansible/+/824566
16:00:44 <choooze> mnasiadka next Wed you mean? or some other time?
16:00:46 <mnasiadka> mgoddard: should we make an update in the reno - that this might cause unplanned instance downtime?
16:01:03 <mgoddard> mnasiadka: no, it was unrelated to libvirt SASL AFAICT
16:01:24 <mgoddard> I will keep investigating
16:01:36 <mnasiadka> choooze: PTG (Project Teams Gathering) is 4-6 April (Kolla-Ansible topics 4-5 April)
16:01:37 <yoctozepto> thanks
16:01:58 <choooze> okaaaay sounds nice
16:02:06 <choooze> thank you guys!
16:03:01 <mnasiadka> choooze: I added an Eventbrite link for the event, make sure you order a virtual ticket ;-)
16:03:26 <mnasiadka> ok, we're over time.
16:03:28 <mnasiadka> Thanks for coming!
16:03:30 <mnasiadka> #endmeeting