15:01:01 #startmeeting kolla 15:01:01 Meeting started Wed Apr 15 15:01:01 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:01:02 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:01:05 The meeting name has been set to 'kolla' 15:01:05 #topic rollcall 15:01:09 \o 15:01:28 o/ 15:01:48 o/ 15:01:50 real crowds 15:01:58 o/ 15:02:00 mnasiadka 15:02:06 o/ 15:02:32 \o 15:03:48 #topic agenda 15:04:00 * Roll-call 15:04:02 * Announcements 15:04:04 * Review action items from last meeting 15:04:06 * CI status 15:04:08 * Kolla Klub https://etherpad.openstack.org/p/kolla-sig 15:04:10 * Ussuri release planning (kayobe) 15:04:12 * Ussuri release planning (kolla & kolla ansible) 15:04:14 * Virtual PTG 15:04:16 * Train release planning (centos 8) 15:04:18 #topic announcements 15:04:56 #info OpenStack now in feature freeze 15:05:10 #info Kolla feature freeze: Apr 27 - May 01 15:05:33 so ~2 weeks to get those features in 15:05:39 Any others 15:05:40 ? 15:06:06 nope 15:06:08 #topic Review action items from last meeting 15:06:10 mgoddard to create a google doc for kolla klub meeting agenda and notes 15:06:12 mgoddard to send reminder to ML about tomorrow's meeting 15:06:14 Both done 15:06:20 oh boy he did 15:06:26 #topic CI status 15:06:46 I think we're pretty green 15:07:17 still the issue with horizon slowness, but no longer blocking CI 15:07:24 priteau: how is kayobe CI now? 15:07:34 Green like my lawn! 15:07:37 does etherpad work for you? 15:08:00 ~o\ 15:08:08 https://etherpad.opendev.org/p/KollaWhiteBoard 15:08:12 priteau: I thought you had decking? :) 15:08:16 gets me only 'loading' 15:08:20 great 15:08:21 yoctozepto: nope, doesn't work for me 15:08:28 priteau: ack, eh 15:08:31 seems I have a cached copy 15:08:32 down for me too 15:08:46 mgoddard: yeah, I refreshed, regretting now ;p 15:08:55 can't updated 15:09:03 ok, we're blind 15:09:23 #topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig 15:09:45 Thanks to those who joined the call, hopefully it was useful 15:10:01 It was a good start 15:10:07 #link https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_4_Sw 15:10:43 #action mgoddard to prepare project update for kolla klub 15:10:43 Hi, I ve re-deployed kolla, any reason why creation of vm instances are failing due to 2020-04-15T15:07:11.496343Z qemu-system-x86_64: failed to initialize KVM: Permission denied ? 15:10:51 born2bake: in a meeting now :-) 15:11:08 born2bake: be back in ~1h 15:11:19 born2bake: and looks like this https://bugs.launchpad.net/kolla/+bug/1681461 15:11:19 Launchpad bug 1681461 in kolla "Instance fails to boot: Could not access KVM kernel module: Permission denied" [High,Confirmed] - Assigned to Chason Chan (chen-xing) 15:11:38 mgoddard: let's move on 15:11:47 yoctozepto: if only we could 15:11:51 Kolla team will discuss ways to gather and maintain operator information, reporting back next time. 15:12:08 mgoddard: ah, sorry 15:12:32 so, what do we track, and how do we track it? 15:13:21 it was not a rhetorical question :) 15:13:59 mgoddard: haha 15:14:08 mgoddard: the basics for sure 15:14:14 mgoddard: are we doing this in doc? 15:14:32 not now 15:14:36 in IRC now 15:14:55 well what are our options for tracking? 15:14:58 etherpad 15:15:00 google doc 15:15:03 wiki 15:15:21 .rst 15:15:43 mgoddard: something doodle-like? 15:15:49 doodle? 15:15:54 isn't that for scheduling? 15:16:05 mgoddard: yeah, that's why I added '-like' 15:16:22 I don't see how it's related 15:16:31 something more organized one can tick they use binary/source centos/ubuntu 15:16:41 which services 15:16:51 maybe we should extend it like a poll to get technical specs for the clouds they operate. 15:16:52 is ceph deployed too 15:17:03 yes, I meant a poll 15:17:07 ceph begone 15:17:11 doodle is a poll for scheduling 15:17:25 I now imagine a tool like debian's popularity-contest 15:17:36 https://packages.debian.org/search?keywords=popularity-contest 15:17:37 oh yeah 15:17:56 but we don't need it automatic for the moment 15:18:08 it should be optional 15:18:09 there are some folks probably willing to spend a few minutes to do the poll 15:18:28 sure, very good idea in the long run 15:18:47 #action osmanlicilegi to adapt popularity-contest for kolla needs 15:18:56 :-) 15:18:59 I see. I was thinking it would be more freeform, but maybe not 15:19:14 yoctozepto: got it :] 15:19:20 it should include something freeform sure 15:19:27 but basics should be easily countable 15:19:47 https://www.surveymonkey.com/ 15:19:52 that kind of thing? 15:20:07 mgoddard: YES! :D 15:20:16 I'll probably be unpopular, but google forms? 15:20:33 at what point are we just duplicating the openstack user survey? 15:20:46 at every 15:20:47 :) 15:20:59 they don't ask kolla specifics? 15:21:05 Etherpad is back up 15:21:14 priteau: yay! :D 15:22:18 HI There, could someone take a look at the following failure: https://zuul.opendev.org/t/openstack/build/69948769a99546d98845ec7995f912f1/log/kolla/build/000_FAILED_cinder-volume.log in relation to the following patch: https://review.opendev.org/#/c/717697/ ? 15:22:19 patch 717697 - kolla (stable/train) - add dpdk_telemetry and logparser for C8, use cento... - 5 patch sets 15:22:36 hi brault, we're currently in a meeting 15:23:02 sorry; 15:23:10 does anyone want to take an action to look into this? 15:23:35 mgoddard: you mean dpdk telemetry? 15:24:01 no, the poll/survey/database 15:24:51 I wished I could but can't promise 15:24:56 really nobody better? 15:25:01 I will take it 15:25:09 thanks! :D 15:25:17 thanks osmanlicilegi 15:25:29 I'm not well with polls but will try to create something for a start 15:25:36 Maybe it would be possible to reuse the openstack user survey framework? to not duplicate things? and store data in weird places? 15:25:36 #action osmanlicilegi to look into kolla klub survey/poll 15:25:39 we can improve later 15:25:50 mnasiadka: that's true 15:26:27 I would suggest for this week osmanlicilegi just looks into available tools, including that used for openstack survey 15:26:44 got it 15:26:45 go get them, osmanlicilegi! 15:27:13 The only surveys I enjoyed taking were implemented with Typeform 15:27:16 #topic Ussuri release planning (kayobe) 15:27:30 And I've created a couple of Typeform surveys, it's easy enough to use 15:27:45 good info pierre 15:27:50 noted typeform 15:27:57 thanks priteau 15:28:21 ok, kayobe 15:28:30 jovial[m] 15:28:53 we have a couple of weeks before freeze 15:29:13 https://review.opendev.org/#/q/project:openstack/kayobe+status:open+branch:master 15:29:20 anything we can squeeze in? 15:29:27 Have we decided a freeze date for kayobe? 15:29:52 same as kolla 15:30:11 Ah, I thought we were trailing from kolla 15:30:17 mnasiadka: you going to update https://review.opendev.org/#/c/709689/ ? 15:30:18 patch 709689 - kayobe - Add framework to deploy user-defined containers on... - 11 patch sets 15:30:36 we have previously trailed, but have been trying to catch up 15:30:38 mgoddard: yes, will do - had other errands to run 15:30:53 Merged openstack/kolla master: nova-compute: add ndctl to expose NVDIMMs to guests https://review.opendev.org/718081 15:30:53 no doubt we'll trail a little but hopefully not much 15:32:14 We can probably merge a good chunk of these. 15:32:22 Lots of small features / bug fixes 15:32:23 quite a few patches open, would be nice to get them moving 15:32:26 yes 15:32:47 I will go through and add RP+1 to some to focus attention 15:33:12 anything else for kayobe? 15:33:43 #topic Ussuri release planning (kolla & kolla ansible) 15:34:33 mgoddard: Nothing else from me 15:34:33 ok, crunch time 15:34:51 looking at https://etherpad.opendev.org/p/KollaWhiteBoard, L187 15:34:51 ;D 15:35:30 mnasiadka: contributor & PTL docs need an update, although it's not affected by feature freezr 15:35:39 mgoddard: on my list 15:35:42 cool 15:36:06 mgoddard: what about docker healthchecks for kolla-ansible - are we going to push it this time, or not? 15:36:35 mnasiadka: I thought that we merged that... 15:36:54 mnasiadka: I'm up for merging it if it gets reviews 15:37:00 hrw: Kolla part - sure 15:37:05 not yet but we should 15:37:20 mgoddard: https://review.opendev.org/#/c/676389/ - I'll rebase it and do RP+1 - let's see if it gets reviews. 15:37:21 patch 676389 - kolla-ansible - Use Docker healthchecks for core services - 49 patch sets 15:37:39 ok 15:37:39 what do they give us apart from being there? :-) I'm for merging because kolla part merged but really... 15:37:41 ah, and incorporate hrw's comments 15:38:01 yoctozepto: well, that's only meaningful for an operator to understand if something is wrong :) 15:38:17 mnasiadka: it's wrong when it's not worky! 15:38:27 yoctozepto: and healthcheck says "not worky!" 15:38:36 I guess if you were monitoring the container state 15:38:43 you could alert on it 15:38:51 yeah, another idea is to do monitoring and alerting 15:39:21 I would say let's merge it - it doesn't hurt anybody - maybe we'll find some good way to integrate it with monasca alerting or something similar 15:39:35 I mean of course after reviews :) 15:39:41 will rebase soon and ask for reviews 15:39:51 mnasiadka: yeah, I'm in 15:40:00 first ovn though, more important, today, I remember 15:40:02 ussuri release... I want to get aarch64 CI for centos8/debian/ubuntu. which now is outside of kolla. for k-a still 2 patches for CI so can have debian/source train->ussuri checks 15:40:10 just going down the list... 15:40:19 [kolla ansible] performance improvements 15:40:29 we merged something for fact caching 15:40:32 it's a start 15:40:38 and then just some tweaks here and there as usual 15:40:43 yay :D 15:40:51 probably needs some docs on ansible tuning 15:40:52 we so performant <3 15:40:55 [kolla ansible] Nova cells v2 15:41:11 I can't see us getting to this, I'll mark it pushed to V 15:41:31 yeah, it's been a tough cycle :/ 15:41:42 [kolla ansible] TLS everywhere 15:41:47 ALMOST! 15:41:48 :D 15:41:49 we merged keystone 15:42:02 there are patches for heat, glance, cinder 15:42:15 I guess it's doable for the core 15:42:23 generalfuzz: if you have patches for other services, would be great to see them 15:43:16 [kolla ansible] More host-level commands (day 2 ops) 15:43:30 Add a new compute host to /etc/hosts 15:43:51 seems we no longer need this as much with the live migration fix 15:44:07 agree 15:44:26 restart would still be nice to have 15:44:41 scottsol: true 15:45:13 [kolla] infra images 15:45:16 pushing to V 15:45:29 [kolla-ansible] Capture Docker logs 15:45:37 mgoddard: unless you run tls because then you need up-to-date name resolution 15:45:45 that said, then you probably should rely on dns... 15:45:54 right 15:46:02 [kolla ansible] OVN support 15:46:06 has 1x +2 15:46:10 OVN doing great 15:46:24 I think it will get another one, it was close to mine before 15:46:30 cool 15:46:43 oven ready 15:46:54 [kolla ansible] Ansible lint 15:47:03 seems we got quite close to this? 15:47:15 it lies somewhere 15:49:47 most of the 'other things' didn't get done 15:50:19 yoctozepto: any thoughts on https://review.opendev.org/#/c/699445/ ? 15:50:20 patch 699445 - kolla-ansible - Add Ansible group check to prechecks - 4 patch sets 15:50:26 Merged openstack/kolla master: gnocchi: install missing python3-wheel package https://review.opendev.org/720159 15:50:32 mgoddard: for ansible-lint I'll check the status of required changes to make it pass and ping you & yoctozepto tomorrow 15:50:37 Merged openstack/kolla-ansible master: Fix Designate not to use etcd coordination backend https://review.opendev.org/719583 15:50:45 mnasiadka: cool 15:50:57 it's not critical - not really a feature :) 15:51:02 mgoddard: was it missing only full coverage afair? 15:51:15 that precheck I mean 15:51:24 yoctozepto: I picked some bad examples in my testing that it couldn't handle 15:52:02 mgoddard: oh well, but it's not making anything worse, right? 15:52:09 no 15:52:29 send mnasiadka at it, and I will re-review later too 15:53:15 ok 15:53:47 still a couple of loose ends for CentOS 8 too, will try to tidy up soon 15:53:51 #topic Virtual PTG 15:54:21 several teams started mentioning vptg 15:54:31 June 1 - 5 15:54:49 I haven't read the latest email about it 15:55:32 but it will be nice to do our PTG at the same time as everyone else again 15:55:42 yep 15:55:45 and I have proper mic now ;D 15:56:10 we will miss donald duck :] 15:56:19 but also know how to switch webcam one from 'Donald Duck' to 'quite normal, sometimes in a bucket' 15:56:46 I think we'll need to decide how many sessions we want 15:57:04 https://www.amazon.com/Trust-GXT-212-Mico-Microphone/dp/B07LC43PXH works good 15:57:21 mgoddard: kolla, kolla-ansible, kayobe, klub? 15:57:45 something like that 15:57:53 klub is probably separate? 15:57:55 I can provide zoom without time limit with potential recording 15:58:06 of course klub members welcome to join in 15:58:20 that would be good 15:58:39 although I think both previous PTGs we tried zoom then switched to meet 15:58:56 osmanlicilegi, hrw: I hardly survived last donald duck session, it was too funny ;D 15:59:04 meet is simple as it seems I think 15:59:04 I use zoom 3 times per week with 3-23 people at same time. 15:59:21 mgoddard: what was the issue? 15:59:25 mgoddard: with zoom 15:59:30 #action mgoddard to read PTG emails and request slots 15:59:34 osmanlicilegi: I am out if gmeet is in use 15:59:35 can't remember 15:59:51 osmanlicilegi: it does not work on my desktop 16:00:02 well, except with hrw, gmeet works best, but it is not usable by Chinese folks 16:00:07 quick final topic 16:00:10 #topic Train release planning (centos 8) 16:00:16 Who added it? 16:00:27 I did 16:00:33 oh why 16:00:56 as we had ussuri release planning in there I would like to see some train release as we have it finally working with c8 16:01:14 makes sense 16:01:51 so we can say 'pip install kolla==9.train.version' and you are free to go 16:02:01 instead of 'fetch head of stable/train' 16:02:25 ok 16:02:33 we have kolla stable/* in stable situation so let release stein/train 16:02:40 acceptance criteria at the bottom of https://etherpad.opendev.org/p/kolla-centos8 16:03:02 needs https://review.opendev.org/717807 16:03:03 patch 717807 - kolla (stable/train) - Add docs and release note for CentOS 8 - 1 patch set 16:03:29 and need to switch from delorean to centos-release-openstack-train 16:03:34 ^ this 16:03:55 also, binary seems brokish 16:03:59 maybe due to delorean 16:04:00 +2+W 16:04:18 yeah 16:04:32 anyone want to pick up delorean? 16:04:50 can do 16:04:57 thanks 16:05:16 #action hrw to switch kolla centos 8 train from delorean to centos-release-openstack-train 16:05:21 (if it exists) 16:05:33 (if not, badger RDO) 16:05:58 ok, we're over time 16:06:10 thanks y'all 16:06:12 #endmeeting