15:00:39 <hrw> #startmeeting kolla
15:00:40 <openstack> Meeting started Wed Jan 20 15:00:39 2021 UTC and is due to finish in 60 minutes.  The chair is hrw. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:41 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:42 <hrw> #chair mgoddard
15:00:44 <openstack> The meeting name has been set to 'kolla'
15:00:45 <openstack> Current chairs: hrw mgoddard
15:01:08 <hrw> long time since I chaired, hope that mgoddard wake up ;D
15:01:13 <hrw> #rollcall
15:01:17 <rafaelweingartne> \o
15:01:18 <mgoddard> o/
15:01:25 <headphoneJames> o/
15:01:28 <hrw> mgoddard: lead, I need coffee
15:01:29 <mgoddard> thanks for starting hrw, was in another
15:01:50 <hrw> no problem
15:03:14 <hrw> #agenda
15:03:22 <hrw> * Roll-call
15:03:22 <hrw> * Announcements
15:03:22 <hrw> * Review action items from the last meeting
15:03:22 <hrw> * CI status
15:03:22 <hrw> * Hot issues
15:03:25 <hrw> ** https://anonymoushash.vmbrasseur.com/2021/01/14/elasticsearch-and-kibana-are-now-business-risks
15:03:28 <hrw> ** Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:03:31 <hrw> ** Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:03:32 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:03:34 <hrw> * Wallaby release planning
15:03:44 <yoctozepto> \o/
15:03:59 <hrw> forgot about #topic ;D
15:04:06 <yoctozepto> doh
15:04:11 <hrw> #topic announcements
15:04:57 <hrw> anyone?
15:04:58 <yoctozepto> want me to chair?
15:05:08 <hrw> #chair yoctozepto
15:05:09 <openstack> Current chairs: hrw mgoddard yoctozepto
15:05:16 <yoctozepto> leadership of USA is changing
15:05:22 <yoctozepto> right today
15:05:31 <hrw> not related to openstack
15:05:33 <yoctozepto> that's pretty important in terms of global politics
15:05:44 <yoctozepto> and openstack is tied to US
15:05:48 <yoctozepto> so close enough! :D
15:05:49 <mgoddard> politics--
15:06:05 <yoctozepto> nobody likes it but it affects us
15:06:06 <yoctozepto> anyhow
15:06:08 <yoctozepto> 3 chairs
15:06:11 <yoctozepto> let's progress :-)
15:06:21 <yoctozepto> #topic Review action items from the last meeting
15:06:52 <yoctozepto> mnasiadka configure registry mirror for kolla CI
15:06:52 <mgoddard> mnasiadka configure registry mirror for kolla CI
15:06:54 <yoctozepto> he did
15:07:00 <yoctozepto> and I backported
15:07:02 <mgoddard> you running this thing yoctozepto?
15:07:08 <yoctozepto> train pending last time I checked
15:07:15 <mgoddard> I'll go grab a beer then :D
15:07:29 <yoctozepto> mgoddard: a'ight, whatever pleases you
15:07:52 <yoctozepto> so all actions completed
15:08:02 <yoctozepto> and watching for results in Train
15:08:14 <yoctozepto> #topic CI status
15:08:46 <yoctozepto> kolla GREEN
15:08:53 <yoctozepto> train pending but should pass soon
15:08:58 <yoctozepto> in fact let's check now
15:09:28 <yoctozepto> ERROR! Unexpected Exception, this is probably a bug: SQLAlchemy 1.3.0 or greater is required.
15:09:30 <yoctozepto> DUH
15:10:02 <hrw> I am running train/ussuri/victoria/master builds on aarch64 to check how we are there as I saw some issues. Expect some patches
15:11:12 <yoctozepto> noted the above error in the whiteboard
15:11:27 <yoctozepto> unless you know right away what the culprit is, let's move onto k-a
15:12:13 <yoctozepto> oh yes, we stopped all the periodics it seems :-)
15:12:38 <yoctozepto> pretty hard to tell if stable branches are affected
15:12:49 <yoctozepto> anyhow, train surely is as that is k-a job that fails
15:12:54 <yoctozepto> (and it's k-a's part of it)
15:13:38 <yoctozepto> master GREEN for sure
15:13:43 <yoctozepto> other stable unknown
15:14:03 <yoctozepto> well, k victoria passed today
15:14:09 <yoctozepto> so that one should be k-a green
15:14:37 <yoctozepto> ussuri yesterday so let's assume too
15:14:40 <yoctozepto> now unto kayobe
15:15:06 <yoctozepto> master looks green
15:15:19 <yoctozepto> added link for victoria
15:15:28 <yoctozepto> and victoria looks green
15:15:52 <yoctozepto> ussuri too
15:15:59 <yoctozepto> train perhaps affected or soon to be affected
15:16:04 <yoctozepto> but so far looking green
15:16:59 <yoctozepto> stein green
15:17:06 <yoctozepto> others unknown; perhaps broken as indicated
15:17:19 <yoctozepto> please remove from the whiteboard if not cared about
15:17:30 <yoctozepto> a'ight that's it for the CI status
15:17:50 <yoctozepto> #topic https://anonymoushash.vmbrasseur.com/2021/01/14/elasticsearch-and-kibana-are-now-business-risks
15:18:00 <yoctozepto> yeah, that's a burning issue
15:18:18 <yoctozepto> we should perhaps pin the versions and issue a communication about it
15:18:37 <mgoddard> I think we're still on 6.x?
15:18:38 * yoctozepto 's interest in Monasca and Venus intensifies
15:19:14 <mgoddard> yeah, ELK 6.x on master
15:19:15 <yoctozepto> mgoddard: on master (Wallaby) too? I thought I have seen a move to es7 but yeah, I remember es6 on stable branches
15:19:20 <yoctozepto> mgoddard: ack, great
15:19:28 <mgoddard> possibly proposed, not merged
15:19:32 <yoctozepto> then communication that it's going to stay at this
15:19:35 <yoctozepto> could be
15:19:53 <hrw> there was es7 proposal
15:20:20 <mgoddard> I guess it should be an opendistro proposal
15:20:29 <mgoddard> whether it's a clean migration, I don't know
15:20:40 <yoctozepto> I can't find it in gerrit
15:22:07 <hrw> https://review.opendev.org/c/openstack/kolla-ansible/+/740986
15:22:14 <yoctozepto> #link https://opendistro.github.io/for-elasticsearch/
15:22:29 <hrw> "Upgrade service configuration for ELK 7"
15:22:48 <yoctozepto> yup, so that I have seen; thanks hrw
15:23:02 <yoctozepto> opendistro looks interesting
15:23:27 <mgoddard> it was pushed by AWS
15:23:38 <mgoddard> the kings of open source
15:23:59 <hrw> 'the kings'
15:24:19 <yoctozepto> fwiw, they include es and kibana, but not logstash; just what we want
15:24:30 <yoctozepto> and they provide the docker images
15:26:48 <mgoddard> commented on the kolla patch
15:26:53 <mgoddard> move on?
15:27:30 <yoctozepto> https://github.com/opendistro/for-elasticsearch-docs/issues/7
15:27:34 <yoctozepto> #link https://github.com/opendistro/for-elasticsearch-docs/issues/7
15:27:43 <yoctozepto> it needs a dump and a restore
15:27:50 <yoctozepto> or so it seems
15:29:21 <yoctozepto> commented too
15:29:24 <yoctozepto> a'ight
15:29:26 <yoctozepto> let's move on
15:29:40 <yoctozepto> #topic Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:29:50 <yoctozepto> how bad is it?
15:30:04 <yoctozepto> it seems it's not that bad after our actions?
15:30:56 <mgoddard> definitely better
15:31:43 <mgoddard> I think we can live with it for a while
15:32:16 <mgoddard> we have two options open, if/when we decide to use them
15:33:27 <yoctozepto> agreed
15:33:33 <yoctozepto> I'll remove this point from the agenda then
15:34:15 <yoctozepto> should we remove the es one too?
15:34:49 <yoctozepto> probably; because it's already handled in the relevant change
15:34:58 <yoctozepto> not much to say about it
15:35:04 <yoctozepto> let's move on
15:35:23 <yoctozepto> #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:35:25 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:35:30 <yoctozepto> i wrote a summary in the whiteboard
15:35:36 <yoctozepto> but did not do much
15:35:38 <yoctozepto> if anything
15:36:14 <yoctozepto> any comments on it?
15:37:07 * mgoddard reading
15:37:17 * yoctozepto waiting for mgoddard to finish reading
15:37:36 <yoctozepto> others are welcome to read as well
15:39:05 <mgoddard> seems like we need to understand those -ing states
15:39:17 <yoctozepto> yup, it was raised by mnasiadka
15:39:27 <yoctozepto> who is not entirely available for a conversation atm
15:40:16 <mnasiadka> well, as long as we don't stop anything while a _working_ agent is doing something with a volume in -ing state, we should be fine.
15:40:40 <mgoddard> is that generally true?
15:40:58 <mgoddard> regardless of this active/passive discussion
15:41:43 <mnasiadka> well, let it do what it needs to do with the volume, and then we can change anything - if we stop it - we might stop it from doing something in the middle of it's job - and it might bring us to some incosistency
15:42:20 <mnasiadka> so my opinion is we need to check for those states during cinder role run, and if it's there - just fail
15:42:27 <yoctozepto> so it's regarding any upgrade whatsoever
15:43:02 <mnasiadka> seems so :)
15:43:16 <mnasiadka> of course we need to remap the volumes to a new agent after we change to cluster
15:43:30 <yoctozepto> they remap themselves
15:43:35 <mnasiadka> oh, cool
15:43:40 <mnasiadka> so just remove the old agent?
15:43:52 <yoctozepto> change config, restart, be happy
15:43:58 <yoctozepto> that's what worked in the CI
15:44:02 <mnasiadka> great
15:44:13 <mnasiadka> so just to be on the safe side - we need a check if any volumes are in -ing state
15:44:15 <yoctozepto> with only -ed states
15:44:23 <yoctozepto> well, we never did
15:44:35 <yoctozepto> it would be a nice addition but we can't help it
15:44:41 <yoctozepto> there could be -ing at any point in time
15:45:02 <yoctozepto> perhaps we should be rolling with disabling agents, waiting and restarting?
15:45:05 <mnasiadka> other option is to not add it, and wait until someone raises a bug (or not)
15:45:12 <yoctozepto> but still, it's orthogonal to active/passive
15:45:56 <mgoddard> is the concern that a volume in an -ing state won't get remapped?
15:46:36 <mnasiadka> rather a concern is that it would be (created|snapshotted|whatever) but with no update to the state in cinder database
15:46:47 <mnasiadka> so when you start a new agent, it does the same thing, and probably fails
15:47:03 <mgoddard> ok
15:47:07 <mgoddard> sounds like a wider issue
15:47:11 <yoctozepto> yup
15:47:29 <yoctozepto> ok, so for active/passive
15:47:44 <yoctozepto> we should perhaps recommend to keep one cinder-volume active?
15:47:53 <yoctozepto> or let users known the tradeoffs involved?
15:48:58 <mgoddard> prechecks would be a good way to tell people their config is broken/risky
15:49:27 <yoctozepto> I_KNOW_WHAT_I_AM_DOING_PLEASE_LET_ME-kind of thing?
15:49:33 <mgoddard> yes
15:50:21 <yoctozepto> and the only active-active that we really support is ceph, no?
15:52:03 <mgoddard> as far as I'm aware
15:52:37 <yoctozepto> well, that's awkward
15:52:40 <yoctozepto> but life is life
15:53:00 <yoctozepto> and then
15:53:08 <yoctozepto> should we impose the coordination now?
15:53:20 <yoctozepto> or only use prechecks here as well
15:54:25 <mgoddard> prechecks
15:56:16 <yoctozepto> and now unto the backend_host
15:56:22 <yoctozepto> I had no time to check that scenario
15:56:27 <yoctozepto> it breaks out of the box
15:56:43 <yoctozepto> if you did not investigate it, then I guess there is not much to talk about
15:56:44 <yoctozepto> :-)
15:57:17 <yoctozepto> #topic Wallaby release planning
15:57:22 <yoctozepto> gogo Wallaby
15:58:23 <headphoneJames> the letsencrypt feature is ready for initial reviews
15:58:27 <headphoneJames> https://review.opendev.org/c/openstack/kolla-ansible/+/741340
15:59:07 <headphoneJames> the test case is still absent, that will be my next bit
15:59:20 <yoctozepto> yay, lovely
15:59:24 <mgoddard> nice, thanks headphoneJames
16:00:29 <headphoneJames> not sure if there is precedence for test cases that mock the service
16:00:32 <hrw> allowed to fail will get requested cleanup and done
16:01:01 <yoctozepto> thanks for your summaries
16:01:08 <yoctozepto> we have to finish
16:01:12 <yoctozepto> thanks all
16:01:15 <yoctozepto> #endmeeting