15:00:39 #startmeeting kolla 15:00:40 Meeting started Wed Jan 20 15:00:39 2021 UTC and is due to finish in 60 minutes. The chair is hrw. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:41 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:42 #chair mgoddard 15:00:44 The meeting name has been set to 'kolla' 15:00:45 Current chairs: hrw mgoddard 15:01:08 long time since I chaired, hope that mgoddard wake up ;D 15:01:13 #rollcall 15:01:17 \o 15:01:18 o/ 15:01:25 o/ 15:01:28 mgoddard: lead, I need coffee 15:01:29 thanks for starting hrw, was in another 15:01:50 no problem 15:03:14 #agenda 15:03:22 * Roll-call 15:03:22 * Announcements 15:03:22 * Review action items from the last meeting 15:03:22 * CI status 15:03:22 * Hot issues 15:03:25 ** https://anonymoushash.vmbrasseur.com/2021/01/14/elasticsearch-and-kibana-are-now-business-risks 15:03:28 ** Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits 15:03:31 ** Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062 15:03:32 Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka) 15:03:34 * Wallaby release planning 15:03:44 \o/ 15:03:59 forgot about #topic ;D 15:04:06 doh 15:04:11 #topic announcements 15:04:57 anyone? 15:04:58 want me to chair? 15:05:08 #chair yoctozepto 15:05:09 Current chairs: hrw mgoddard yoctozepto 15:05:16 leadership of USA is changing 15:05:22 right today 15:05:31 not related to openstack 15:05:33 that's pretty important in terms of global politics 15:05:44 and openstack is tied to US 15:05:48 so close enough! :D 15:05:49 politics-- 15:06:05 nobody likes it but it affects us 15:06:06 anyhow 15:06:08 3 chairs 15:06:11 let's progress :-) 15:06:21 #topic Review action items from the last meeting 15:06:52 mnasiadka configure registry mirror for kolla CI 15:06:52 mnasiadka configure registry mirror for kolla CI 15:06:54 he did 15:07:00 and I backported 15:07:02 you running this thing yoctozepto? 15:07:08 train pending last time I checked 15:07:15 I'll go grab a beer then :D 15:07:29 mgoddard: a'ight, whatever pleases you 15:07:52 so all actions completed 15:08:02 and watching for results in Train 15:08:14 #topic CI status 15:08:46 kolla GREEN 15:08:53 train pending but should pass soon 15:08:58 in fact let's check now 15:09:28 ERROR! Unexpected Exception, this is probably a bug: SQLAlchemy 1.3.0 or greater is required. 15:09:30 DUH 15:10:02 I am running train/ussuri/victoria/master builds on aarch64 to check how we are there as I saw some issues. Expect some patches 15:11:12 noted the above error in the whiteboard 15:11:27 unless you know right away what the culprit is, let's move onto k-a 15:12:13 oh yes, we stopped all the periodics it seems :-) 15:12:38 pretty hard to tell if stable branches are affected 15:12:49 anyhow, train surely is as that is k-a job that fails 15:12:54 (and it's k-a's part of it) 15:13:38 master GREEN for sure 15:13:43 other stable unknown 15:14:03 well, k victoria passed today 15:14:09 so that one should be k-a green 15:14:37 ussuri yesterday so let's assume too 15:14:40 now unto kayobe 15:15:06 master looks green 15:15:19 added link for victoria 15:15:28 and victoria looks green 15:15:52 ussuri too 15:15:59 train perhaps affected or soon to be affected 15:16:04 but so far looking green 15:16:59 stein green 15:17:06 others unknown; perhaps broken as indicated 15:17:19 please remove from the whiteboard if not cared about 15:17:30 a'ight that's it for the CI status 15:17:50 #topic https://anonymoushash.vmbrasseur.com/2021/01/14/elasticsearch-and-kibana-are-now-business-risks 15:18:00 yeah, that's a burning issue 15:18:18 we should perhaps pin the versions and issue a communication about it 15:18:37 I think we're still on 6.x? 15:18:38 * yoctozepto 's interest in Monasca and Venus intensifies 15:19:14 yeah, ELK 6.x on master 15:19:15 mgoddard: on master (Wallaby) too? I thought I have seen a move to es7 but yeah, I remember es6 on stable branches 15:19:20 mgoddard: ack, great 15:19:28 possibly proposed, not merged 15:19:32 then communication that it's going to stay at this 15:19:35 could be 15:19:53 there was es7 proposal 15:20:20 I guess it should be an opendistro proposal 15:20:29 whether it's a clean migration, I don't know 15:20:40 I can't find it in gerrit 15:22:07 https://review.opendev.org/c/openstack/kolla-ansible/+/740986 15:22:14 #link https://opendistro.github.io/for-elasticsearch/ 15:22:29 "Upgrade service configuration for ELK 7" 15:22:48 yup, so that I have seen; thanks hrw 15:23:02 opendistro looks interesting 15:23:27 it was pushed by AWS 15:23:38 the kings of open source 15:23:59 'the kings' 15:24:19 fwiw, they include es and kibana, but not logstash; just what we want 15:24:30 and they provide the docker images 15:26:48 commented on the kolla patch 15:26:53 move on? 15:27:30 https://github.com/opendistro/for-elasticsearch-docs/issues/7 15:27:34 #link https://github.com/opendistro/for-elasticsearch-docs/issues/7 15:27:43 it needs a dump and a restore 15:27:50 or so it seems 15:29:21 commented too 15:29:24 a'ight 15:29:26 let's move on 15:29:40 #topic Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits 15:29:50 how bad is it? 15:30:04 it seems it's not that bad after our actions? 15:30:56 definitely better 15:31:43 I think we can live with it for a while 15:32:16 we have two options open, if/when we decide to use them 15:33:27 agreed 15:33:33 I'll remove this point from the agenda then 15:34:15 should we remove the es one too? 15:34:49 probably; because it's already handled in the relevant change 15:34:58 not much to say about it 15:35:04 let's move on 15:35:23 #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062 15:35:25 Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka) 15:35:30 i wrote a summary in the whiteboard 15:35:36 but did not do much 15:35:38 if anything 15:36:14 any comments on it? 15:37:07 * mgoddard reading 15:37:17 * yoctozepto waiting for mgoddard to finish reading 15:37:36 others are welcome to read as well 15:39:05 seems like we need to understand those -ing states 15:39:17 yup, it was raised by mnasiadka 15:39:27 who is not entirely available for a conversation atm 15:40:16 well, as long as we don't stop anything while a _working_ agent is doing something with a volume in -ing state, we should be fine. 15:40:40 is that generally true? 15:40:58 regardless of this active/passive discussion 15:41:43 well, let it do what it needs to do with the volume, and then we can change anything - if we stop it - we might stop it from doing something in the middle of it's job - and it might bring us to some incosistency 15:42:20 so my opinion is we need to check for those states during cinder role run, and if it's there - just fail 15:42:27 so it's regarding any upgrade whatsoever 15:43:02 seems so :) 15:43:16 of course we need to remap the volumes to a new agent after we change to cluster 15:43:30 they remap themselves 15:43:35 oh, cool 15:43:40 so just remove the old agent? 15:43:52 change config, restart, be happy 15:43:58 that's what worked in the CI 15:44:02 great 15:44:13 so just to be on the safe side - we need a check if any volumes are in -ing state 15:44:15 with only -ed states 15:44:23 well, we never did 15:44:35 it would be a nice addition but we can't help it 15:44:41 there could be -ing at any point in time 15:45:02 perhaps we should be rolling with disabling agents, waiting and restarting? 15:45:05 other option is to not add it, and wait until someone raises a bug (or not) 15:45:12 but still, it's orthogonal to active/passive 15:45:56 is the concern that a volume in an -ing state won't get remapped? 15:46:36 rather a concern is that it would be (created|snapshotted|whatever) but with no update to the state in cinder database 15:46:47 so when you start a new agent, it does the same thing, and probably fails 15:47:03 ok 15:47:07 sounds like a wider issue 15:47:11 yup 15:47:29 ok, so for active/passive 15:47:44 we should perhaps recommend to keep one cinder-volume active? 15:47:53 or let users known the tradeoffs involved? 15:48:58 prechecks would be a good way to tell people their config is broken/risky 15:49:27 I_KNOW_WHAT_I_AM_DOING_PLEASE_LET_ME-kind of thing? 15:49:33 yes 15:50:21 and the only active-active that we really support is ceph, no? 15:52:03 as far as I'm aware 15:52:37 well, that's awkward 15:52:40 but life is life 15:53:00 and then 15:53:08 should we impose the coordination now? 15:53:20 or only use prechecks here as well 15:54:25 prechecks 15:56:16 and now unto the backend_host 15:56:22 I had no time to check that scenario 15:56:27 it breaks out of the box 15:56:43 if you did not investigate it, then I guess there is not much to talk about 15:56:44 :-) 15:57:17 #topic Wallaby release planning 15:57:22 gogo Wallaby 15:58:23 the letsencrypt feature is ready for initial reviews 15:58:27 https://review.opendev.org/c/openstack/kolla-ansible/+/741340 15:59:07 the test case is still absent, that will be my next bit 15:59:20 yay, lovely 15:59:24 nice, thanks headphoneJames 16:00:29 not sure if there is precedence for test cases that mock the service 16:00:32 allowed to fail will get requested cleanup and done 16:01:01 thanks for your summaries 16:01:08 we have to finish 16:01:12 thanks all 16:01:15 #endmeeting