15:00:14 <yoctozepto> #startmeeting kolla
15:00:14 <hrw> yoctozepto: #startmeeting?
15:00:15 <openstack> Meeting started Wed Jul  8 15:00:14 2020 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <openstack> The meeting name has been set to 'kolla'
15:00:23 <yoctozepto> hrw: you guessed it
15:00:29 <yoctozepto> #topic rollcall
15:00:35 <hrw> o~
15:00:38 <mnasiadka> o/
15:00:41 <osmanlicilegi> o/
15:00:46 <yoctozepto> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak - meeting ongoing
15:00:59 <yoctozepto> \o/
15:01:07 <yoctozepto> \o\
15:01:12 <yoctozepto> /o/
15:01:58 <admin0> \o
15:02:24 <jovial[m]> o/
15:02:47 <yoctozepto> priteau, dougsz: meeting, meeting?
15:03:15 <yoctozepto> #topic agenda
15:03:22 <yoctozepto> * Roll-call
15:03:22 <yoctozepto> * Announcements
15:03:22 <yoctozepto> * Review action items from last meeting
15:03:22 <yoctozepto> * CI status
15:03:22 <yoctozepto> * Keystone fernet key sync followup https://review.opendev.org/707080
15:03:22 <yoctozepto> * Victoria release planning (kayobe)
15:03:22 <yoctozepto> * Victoria release planning (kolla & kolla ansible)
15:03:23 <patchbot> patch 707080 - kolla-ansible - Fix fernet bootstrap and key distribution - follow up - 14 patch sets
15:03:23 <yoctozepto> * Kolla klub
15:03:43 <yoctozepto> #topic Announcements
15:03:44 <bbezak> o/
15:04:08 <yoctozepto> #info The first Kolla Kall is happening tomorrow
15:04:13 <yoctozepto> #link https://wiki.openstack.org/wiki/Meetings/Kolla/Kall
15:04:35 <priteau> yoctozepto: sorry I won't be available for the meeting today
15:04:43 <yoctozepto> it is open to all interested in developing kolla and friends
15:04:51 <jovial[m]> I assume you will only be discussing kolla and not kayobe?
15:04:52 <yoctozepto> priteau: ack, no probies
15:05:04 <openstackgerrit> gugug proposed openstack/kolla-ansible master: Remove the planned merge_configs  https://review.opendev.org/739525
15:05:06 <yoctozepto> jovial[m]: kayobe too
15:05:18 <yoctozepto> in the agenda we have the documentation review
15:05:36 <yoctozepto> it is very likely kolla docs will take up enough time to fail to fit kayobe in this session though
15:05:38 <yoctozepto> but we will try
15:05:39 <jovial[m]> awesome, thanks :)
15:05:48 <mnasiadka> docs review? I just found out I can't join! ;)
15:05:56 <yoctozepto> mnasiadka: ha-ha
15:06:03 <yoctozepto> jovial[m]: :-)
15:06:18 <yoctozepto> anything else to share as announcement?
15:06:25 <mnasiadka> yoctozepto: kayobe also needs docs review I guess :)
15:06:32 <yoctozepto> any new kolla babies? (in progress or no)
15:06:43 <yoctozepto> mnasiadka: yes, it is scheduled in the agenda
15:06:55 <yoctozepto> but I am worried we will eat up enough time for the previous two
15:07:14 <yoctozepto> the Kall happens every other week so the next will be in two weeks time
15:07:20 <yoctozepto> it alternates with Klub
15:07:32 <yoctozepto> ok, I assume no announcements
15:07:34 <yoctozepto> let's go
15:07:42 <yoctozepto> #topic Review action items from last meeting
15:07:53 <yoctozepto> hrw to switch kolla to use stable branches and update release documentation
15:08:14 <yoctozepto> I did not notice but I was a bit overwhelmed by other things
15:08:30 <openstackgerrit> Doug Szumski proposed openstack/kolla-ansible stable/train: Use kolla_logs volume for Elasticsearch  https://review.opendev.org/740015
15:08:41 <mnasiadka> there is a change backport to stable/ussuri - but haven't seen release docs update
15:08:47 <yoctozepto> it looks started https://review.opendev.org/738907
15:08:48 <patchbot> patch 738907 - kolla (stable/ussuri) - version-check: handle using of stable branch tarballs - 2 patch sets
15:09:12 <yoctozepto> "cherry-picked from master + updated SOURCES in config.py"
15:09:21 <yoctozepto> it seems gerrit lost the connection between the two
15:09:24 <hrw> yoctozepto: patch for stable in gerrit
15:09:49 <yoctozepto> https://review.opendev.org/738906
15:09:49 <patchbot> patch 738906 - kolla - version-check: handle using of stable branch tarballs - 3 patch sets
15:09:55 <hrw> first we need to merge one into master
15:10:01 <yoctozepto> #link https://review.opendev.org/738906
15:10:02 <patchbot> patch 738906 - kolla - version-check: handle using of stable branch tarballs - 3 patch sets
15:10:02 <hrw> ussuri one was CI test more
15:10:10 <yoctozepto> ack, that makes sense
15:10:52 <yoctozepto> no other tasks
15:11:03 <yoctozepto> I spent some time with timesync and family but did not finish
15:11:11 <yoctozepto> duties, duties
15:11:23 <yoctozepto> moving on
15:11:27 <yoctozepto> #topic CI status
15:11:45 <hrw> rocky is RED still
15:11:51 <yoctozepto> yeah, but rocky is em
15:11:58 <yoctozepto> yet we want it fixed for other reasons
15:12:09 <hrw> and we have fix in gerrit which fails anyway
15:12:09 <yoctozepto> anyhow, it generally looks green
15:12:22 <yoctozepto> the timeouts are annoying though
15:12:25 <hrw> yep
15:12:30 <yoctozepto> the horizon slowness is ---
15:12:30 <hrw> and aarch64 issues
15:12:42 <mgoddard> o/
15:12:44 <mgoddard> sorry I'm late, thanks for getting started
15:12:59 <yoctozepto> #chair mgoddard
15:13:00 <openstack> Current chairs: mgoddard yoctozepto
15:13:05 <yoctozepto> welcome mgoddard
15:13:14 <yoctozepto> #topic Keystone fernet key sync followup https://review.opendev.org/707080
15:13:14 <patchbot> patch 707080 - kolla-ansible - Fix fernet bootstrap and key distribution - follow up - 14 patch sets
15:13:22 <yoctozepto> mnasiadka, the floor is yours
15:13:48 <mnasiadka> I'm afraid I didn't have time to even look in that change, just a quick peek
15:14:07 <yoctozepto> mnasiadka: so nothing to add? sad :-(
15:14:28 <yoctozepto> but I obviously understand, in a similar position
15:14:33 <dougsz> \o
15:14:50 <mgoddard> I did another review of it
15:15:04 <mnasiadka> yoctozepto: I'll spawn some VMs on Friday and try to fix it though - if there's nobody that can pick it up
15:15:42 <mnasiadka> mgoddard: saw that, thanks - but I think that change needs a major rework, I just left it in some weird state - might be easier to have a fresh approach
15:16:12 <yoctozepto> mnasiadka: I only thought you would be the best guy to continue this, already told you that
15:16:13 <mgoddard> I was wondering if there is some part of it we can merge to make an improvement, without looking for perfection
15:16:27 <yoctozepto> but if you strongly disagree, then we can reorganize it
15:16:29 <yoctozepto> mgoddard: ++
15:17:04 <yoctozepto> we seem to have made some parts of the process better but worsened others, could limit to reversing the worsening and obvious wrongs
15:17:13 <mnasiadka> tomorrow I'm pretty busy, but I'll move this to top of my priorities list, so at least I can look into what we could split out of that change
15:17:17 <yoctozepto> but I guess it does not end up being too far away from perfection then
15:17:25 <yoctozepto> mnasiadka: sounds great to me
15:17:36 <mgoddard> another thing - should keystone wait for keys on startup, or just fail and retry at the docker level?
15:17:38 <mnasiadka> yoctozepto: fernet won't change approach in keystone, so we just need to make it semi-good :)
15:17:53 <yoctozepto> mnasiadka: haha, that true
15:18:14 <yoctozepto> quasi-good would be probably even more appropriate
15:18:25 <mnasiadka> mgoddard: retrying at Docker level might be good enough
15:18:37 <mgoddard> it's more visible
15:18:57 <mgoddard> and we should be able to restructure the ansible to wait for keys to be distributed
15:19:10 <mgoddard> or actively sync in handlers
15:19:32 <mnasiadka> mgoddard: good idea, let me look into that on Friday - tomorrow I'm a bit busy
15:19:49 <mnasiadka> and we can discuss on the channel
15:20:10 <mgoddard> maybe a handler like: include_tasks: distribute_fernet.yml
15:20:27 <mgoddard> or maybe not
15:20:57 <mgoddard> it could work
15:21:18 <yoctozepto> anyhow, as far as chrony/ntp/timesync is concerned I did some analysis that I share with infra and you might be interested in when debugging https://etherpad.opendev.org/p/ci-timesync
15:21:22 <yoctozepto> #link https://etherpad.opendev.org/p/ci-timesync
15:21:33 <yoctozepto> (side note I forgot to share)
15:21:45 <openstackgerrit> jacky06 proposed openstack/kolla-ansible master: Remove the waiting for ironic-api to be accessible  https://review.opendev.org/740018
15:22:38 <mnasiadka> yoctozepto: regarding timesync I'm really unhappy that we merged a change, that uninstalls chrony on the OS level without giving the option for not doing so :)
15:22:59 <yoctozepto> mnasiadka: how so? it is controlled by enable_chrony
15:23:46 <mnasiadka> yoctozepto: yeah, still hard to override it on host/group level :)
15:24:14 <mnasiadka> yoctozepto: anyway, just laughing, it's a bit irritating to see that, if you don't track changes
15:24:15 <mgoddard> logically it makes sense, but in practice might cause issues
15:24:34 <mgoddard> you could use the chrony group to control where it lands
15:25:59 <yoctozepto> anyhow, we agreed to deprecate the switch and switch it to False by default
15:26:09 <yoctozepto> we don't recommend containerizing chrony
15:26:14 <yoctozepto> (unless we do)
15:27:09 <mgoddard> so from your notes yoctozepto it looks like ntpd is slow to sync, but chrony and timesyncd are better
15:27:20 <yoctozepto> in their current infra config
15:27:36 <mgoddard> often with ntpd it's combined with a one time sync
15:27:45 <yoctozepto> it probably failed
15:27:56 <mgoddard> maybe newer daemons have some better startup logic
15:27:57 <yoctozepto> anyway, infra will likely switch to the distro defaults
15:28:08 <yoctozepto> ntpd is no longer a default anywhere
15:28:10 <mnasiadka> so bye bye ntpd
15:29:22 <yoctozepto> more or less, yes
15:30:11 <mgoddard> shall we move on?
15:30:16 <mgoddard> #topic Victoria release planning (kayobe)
15:30:42 <mgoddard> #link https://etherpad.opendev.org/p/KollaWhiteBoard
15:30:57 <mgoddard> L292
15:31:00 <mgoddard> jovial[m]
15:31:16 <jovial[m]> hey
15:31:21 <mgoddard> dougsz
15:31:32 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: Remove the planned merge_configs  https://review.opendev.org/739525
15:31:44 <dougsz> hi there
15:31:44 <mgoddard> is anyone working on victoria features yet?
15:31:51 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: Remove the ml2_conf.ini merging for agents  https://review.opendev.org/739525
15:32:05 <dougsz> not I
15:32:13 <jovial[m]> priteau: was working on multiple environments, right?
15:32:21 <mgoddard> yes, there was a patch
15:32:23 <priteau> I’ve not been able to work on multiple environments recently.
15:32:46 <mgoddard> https://review.opendev.org/734867
15:32:47 <patchbot> patch 734867 - kayobe - Support multiple environments from a single config... - 1 patch set
15:32:53 <priteau> Will resume as soon as I’ve finished the project keeping me busy
15:33:54 <mgoddard> I should give it a review
15:34:45 <mgoddard> https://review.opendev.org/#/q/project:openstack/kayobe+status:open
15:34:55 <priteau> It’s not fully done yet but happy to have initial comments
15:35:07 <mgoddard> we should pick a few patches to prioritise to keep on top of reviews
15:35:19 <jovial[m]> sounds good to me
15:35:22 <mgoddard> backports of course
15:35:56 <mgoddard> everyone pick a (non-backport) patch and add RP+1 :)
15:36:38 <admin0> how to ask kolla ansible to use nfs for glance
15:37:30 <admin0> i see the docs saying glance_file_datadir_volume  .. is that done on the controllers ?
15:37:47 <mgoddard> admin0: we're in a meeting at the moment
15:37:52 <mgoddard> call back in 25 minutes
15:37:53 <admin0> oh .. sorry
15:37:57 <mgoddard> any other kayobe things?
15:37:57 <admin0> carry on
15:38:35 <dougsz> hmm, nothing I can think of
15:38:41 <priteau> I’ve pushed minor changes to the train image regex please review
15:38:48 <mgoddard> thanks
15:39:04 <mgoddard> #topic Victoria release planning (kolla & kolla ansible)
15:39:26 <mgoddard> whiteboard L178+
15:40:00 <mgoddard> good to see hrw working hard on infra images, ubuntu focal and stable branch lifecycle
15:40:01 <hrw> yo
15:40:17 <hrw> focal needs td-agent4 and novajoin fixes to merge
15:40:19 <mgoddard> please review his patches to keep him interested :)
15:40:30 <hrw> then it will work for both x86 and aarch64
15:40:37 <hrw> s/work/build
15:41:10 <mgoddard> ok. Let's keep an eye on fluentd logs for any issues
15:41:11 <hrw> I can add RP+1 if someone needs more pinging
15:41:15 <yoctozepto> the good news in this regard is that I need designate in one of my projects so it is easier to fit it in my schedule
15:41:39 <mgoddard> nice
15:42:18 <mgoddard> hrw: how does td-agent map to fluentd version?
15:42:31 <hrw> mgoddard: have to check
15:42:50 <dougsz> haven't seen a stable fluentd release using td4 yet
15:42:58 <hrw> mgoddard: 'latest v1'
15:43:03 <hrw> dougsz: they did it last week
15:43:16 <hrw> dougsz: https://review.opendev.org/739935
15:43:16 <patchbot> patch 739935 - kolla - fluentd: migrate to td-agent 4 where possible - 3 patch sets
15:43:48 <dougsz> yeah, but that's not a stable fluentd release
15:44:23 <hrw> dougsz: I wanted to respond with 'since when td-agent is sane' but this is logged
15:44:28 <hrw> ;D
15:44:46 <hrw> dougsz: if we want focal then either td-agent4 or fluentd from gem
15:44:51 <hrw> choose your poison
15:45:17 <mgoddard> do we support fluentd on PPC?
15:45:34 <mgoddard> 'support'
15:45:35 <dougsz> perhaps we should ask them what the plans are for the next stable release
15:45:38 <hrw> I would go with just fluentd everywhere but I am outvoted on 'just use distro/foss instead of random binaries'
15:45:50 <hrw> mgoddard: do we support ppc? we just have basic support
15:46:21 <hrw> dougsz: go and ask. I prefer not to. they barely made 4.0.0 release
15:46:33 <dougsz> :) OK, I will ask
15:46:38 * hrw ends bashing and wait for next topic
15:46:58 <mgoddard> Any other features we should discuss?
15:47:02 <mnasiadka> dougsz: first read hrw's history of asking
15:47:20 <mgoddard> I've been pushing patches for performance improvements, reviews appreciated
15:47:22 <hrw> dougsz: https://github.com/treasure-data/omnibus-td-agent/issues/209 is fun read
15:47:43 <mnasiadka> dougsz: since when fluentd has stable releases? :)
15:47:53 <hrw> mgoddard: your patches remind me to finally learn ansible..
15:48:12 <mgoddard> hrw: you should! it's really not hard
15:49:04 <hrw> ;)
15:49:23 <mgoddard> #topic Open discussion
15:49:48 <hrw> anyone wants to say something?
15:50:37 <mgoddard> seems not
15:50:42 <mgoddard> Let's finish early
15:50:45 <mnasiadka> I revived docker healtchecks on kolla-ansible - would be happy if somebody reviewed
15:50:50 <mgoddard> Thanks everyone, and thanks yoctozepto for chairing
15:50:53 <mgoddard> #endmeeting