15:01:01 <mgoddard> #startmeeting kolla
15:01:01 <openstack> Meeting started Wed Apr 15 15:01:01 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:02 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:05 <openstack> The meeting name has been set to 'kolla'
15:01:05 <mgoddard> #topic rollcall
15:01:09 <mgoddard> \o
15:01:28 <yoctozepto> o/
15:01:48 <scottsol> o/
15:01:50 <yoctozepto> real crowds
15:01:58 <osmanlicilegi> o/
15:02:00 <yoctozepto> mnasiadka
15:02:06 <mnasiadka> o/
15:02:32 <priteau> \o
15:03:48 <mgoddard> #topic agenda
15:04:00 <mgoddard> * Roll-call
15:04:02 <mgoddard> * Announcements
15:04:04 <mgoddard> * Review action items from last meeting
15:04:06 <mgoddard> * CI status
15:04:08 <mgoddard> * Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:04:10 <mgoddard> * Ussuri release planning (kayobe)
15:04:12 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:04:14 <mgoddard> * Virtual PTG
15:04:16 <mgoddard> * Train release planning (centos 8)
15:04:18 <mgoddard> #topic announcements
15:04:56 <mgoddard> #info OpenStack now in feature freeze
15:05:10 <mgoddard> #info Kolla feature freeze: Apr 27 - May 01
15:05:33 <mgoddard> so ~2 weeks to get those features in
15:05:39 <mgoddard> Any others
15:05:40 <mgoddard> ?
15:06:06 <osmanlicilegi> nope
15:06:08 <mgoddard> #topic Review action items from last meeting
15:06:10 <mgoddard> mgoddard to create a google doc for kolla klub meeting agenda and notes
15:06:12 <mgoddard> mgoddard to send reminder to ML about tomorrow's meeting
15:06:14 <mgoddard> Both done
15:06:20 <yoctozepto> oh boy he did
15:06:26 <mgoddard> #topic CI status
15:06:46 <mgoddard> I think we're pretty green
15:07:17 <mgoddard> still the issue with horizon slowness, but no longer blocking CI
15:07:24 <mgoddard> priteau: how is kayobe CI now?
15:07:34 <priteau> Green like my lawn!
15:07:37 <yoctozepto> does etherpad work for you?
15:08:00 <hrw> ~o\
15:08:08 <yoctozepto> https://etherpad.opendev.org/p/KollaWhiteBoard
15:08:12 <mgoddard> priteau: I thought you had decking? :)
15:08:16 <yoctozepto> gets me only 'loading'
15:08:20 <mgoddard> great
15:08:21 <priteau> yoctozepto: nope, doesn't work for me
15:08:28 <yoctozepto> priteau: ack, eh
15:08:31 <mgoddard> seems I have a cached copy
15:08:32 <osmanlicilegi> down for me too
15:08:46 <yoctozepto> mgoddard: yeah, I refreshed, regretting now ;p
15:08:55 <mgoddard> can't updated
15:09:03 <mgoddard> ok, we're blind
15:09:23 <mgoddard> #topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig
15:09:45 <mgoddard> Thanks to those who joined the call, hopefully it was useful
15:10:01 <osmanlicilegi> It was a good start
15:10:07 <mgoddard> #link https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_4_Sw
15:10:43 <mgoddard> #action mgoddard to prepare project update for kolla klub
15:10:43 <born2bake> Hi, I ve re-deployed kolla, any reason why creation of vm instances are failing due to 2020-04-15T15:07:11.496343Z qemu-system-x86_64: failed to initialize KVM: Permission denied ?
15:10:51 <yoctozepto> born2bake: in a meeting now :-)
15:11:08 <hrw> born2bake: be back in ~1h
15:11:19 <yoctozepto> born2bake: and looks like this https://bugs.launchpad.net/kolla/+bug/1681461
15:11:19 <openstack> Launchpad bug 1681461 in kolla "Instance fails to boot: Could not access KVM kernel module: Permission denied" [High,Confirmed] - Assigned to Chason Chan (chen-xing)
15:11:38 <yoctozepto> mgoddard: let's move on
15:11:47 <mgoddard> yoctozepto: if only we could
15:11:51 <mgoddard> Kolla team will discuss ways to gather and maintain operator information, reporting back next time.
15:12:08 <yoctozepto> mgoddard: ah, sorry
15:12:32 <mgoddard> so, what do we track, and how do we track it?
15:13:21 <mgoddard> it was not a rhetorical question :)
15:13:59 <yoctozepto> mgoddard: haha
15:14:08 <yoctozepto> mgoddard: the basics for sure
15:14:14 <yoctozepto> mgoddard: are we doing this in doc?
15:14:32 <mgoddard> not now
15:14:36 <mgoddard> in IRC now
15:14:55 <mgoddard> well what are our options for tracking?
15:14:58 <mgoddard> etherpad
15:15:00 <mgoddard> google doc
15:15:03 <mgoddard> wiki
15:15:21 <mgoddard> .rst
15:15:43 <yoctozepto> mgoddard: something doodle-like?
15:15:49 <mgoddard> doodle?
15:15:54 <mgoddard> isn't that for scheduling?
15:16:05 <yoctozepto> mgoddard: yeah, that's why I added '-like'
15:16:22 <mgoddard> I don't see how it's related
15:16:31 <yoctozepto> something more organized one can tick they use binary/source centos/ubuntu
15:16:41 <yoctozepto> which services
15:16:51 <osmanlicilegi> maybe we should extend it like a poll to get technical specs for the clouds they operate.
15:16:52 <yoctozepto> is ceph deployed too
15:17:03 <yoctozepto> yes, I meant a poll
15:17:07 <mnasiadka> ceph begone
15:17:11 <yoctozepto> doodle is a poll for scheduling
15:17:25 <osmanlicilegi> I now imagine a tool like debian's popularity-contest
15:17:36 <osmanlicilegi> https://packages.debian.org/search?keywords=popularity-contest
15:17:37 <yoctozepto> oh yeah
15:17:56 <yoctozepto> but we don't need it automatic for the moment
15:18:08 <osmanlicilegi> it should be optional
15:18:09 <yoctozepto> there are some folks probably willing to spend a few minutes to do the poll
15:18:28 <yoctozepto> sure, very good idea in the long run
15:18:47 <yoctozepto> #action osmanlicilegi to adapt popularity-contest for kolla needs
15:18:56 <yoctozepto> :-)
15:18:59 <mgoddard> I see. I was thinking it would be more freeform, but maybe not
15:19:14 <osmanlicilegi> yoctozepto: got it :]
15:19:20 <yoctozepto> it should include something freeform sure
15:19:27 <yoctozepto> but basics should be easily countable
15:19:47 <mgoddard> https://www.surveymonkey.com/
15:19:52 <mgoddard> that kind of thing?
15:20:07 <yoctozepto> mgoddard: YES! :D
15:20:16 <mnasiadka> I'll probably be unpopular, but google forms?
15:20:33 <mgoddard> at what point are we just duplicating the openstack user survey?
15:20:46 <mnasiadka> at every
15:20:47 <mnasiadka> :)
15:20:59 <yoctozepto> they don't ask kolla specifics?
15:21:05 <priteau> Etherpad is back up
15:21:14 <yoctozepto> priteau: yay! :D
15:22:18 <brault> HI There, could someone take a look at the following failure:  https://zuul.opendev.org/t/openstack/build/69948769a99546d98845ec7995f912f1/log/kolla/build/000_FAILED_cinder-volume.log in relation to the following patch: https://review.opendev.org/#/c/717697/ ?
15:22:19 <patchbot> patch 717697 - kolla (stable/train) - add dpdk_telemetry and logparser for C8, use cento... - 5 patch sets
15:22:36 <mgoddard> hi brault, we're currently in a meeting
15:23:02 <brault> sorry;
15:23:10 <mgoddard> does anyone want to take an action to look into this?
15:23:35 <yoctozepto> mgoddard: you mean dpdk telemetry?
15:24:01 <mgoddard> no, the poll/survey/database
15:24:51 <yoctozepto> I wished I could but can't promise
15:24:56 <yoctozepto> really nobody better?
15:25:01 <osmanlicilegi> I will take it
15:25:09 <yoctozepto> thanks! :D
15:25:17 <mgoddard> thanks osmanlicilegi
15:25:29 <osmanlicilegi> I'm not well with polls but will try to create something for a start
15:25:36 <mnasiadka> Maybe it would be possible to reuse the openstack user survey framework? to not duplicate things? and store data in weird places?
15:25:36 <mgoddard> #action osmanlicilegi to look into kolla klub survey/poll
15:25:39 <osmanlicilegi> we can improve later
15:25:50 <mgoddard> mnasiadka: that's true
15:26:27 <mgoddard> I would suggest for this week osmanlicilegi just looks into available tools, including that used for openstack survey
15:26:44 <osmanlicilegi> got it
15:26:45 <yoctozepto> go get them, osmanlicilegi!
15:27:13 <priteau> The only surveys I enjoyed taking were implemented with Typeform
15:27:16 <mgoddard> #topic Ussuri release planning (kayobe)
15:27:30 <priteau> And I've created a couple of Typeform surveys, it's easy enough to use
15:27:45 <mgoddard> good info pierre
15:27:50 <osmanlicilegi> noted typeform
15:27:57 <osmanlicilegi> thanks priteau
15:28:21 <mgoddard> ok, kayobe
15:28:30 <mgoddard> jovial[m]
15:28:53 <mgoddard> we have a couple of weeks before freeze
15:29:13 <mgoddard> https://review.opendev.org/#/q/project:openstack/kayobe+status:open+branch:master
15:29:20 <jovial[m]> anything we can squeeze in?
15:29:27 <priteau> Have we decided a freeze date for kayobe?
15:29:52 <mgoddard> same as kolla
15:30:11 <priteau> Ah, I thought we were trailing from kolla
15:30:17 <mgoddard> mnasiadka: you going to update https://review.opendev.org/#/c/709689/ ?
15:30:18 <patchbot> patch 709689 - kayobe - Add framework to deploy user-defined containers on... - 11 patch sets
15:30:36 <mgoddard> we have previously trailed, but have been trying to catch up
15:30:38 <mnasiadka> mgoddard: yes, will do - had other errands to run
15:30:53 <openstackgerrit> Merged openstack/kolla master: nova-compute: add ndctl to expose NVDIMMs to guests  https://review.opendev.org/718081
15:30:53 <mgoddard> no doubt we'll trail a little but hopefully not much
15:32:14 <priteau> We can probably merge a good chunk of these.
15:32:22 <priteau> Lots of small features / bug fixes
15:32:23 <mgoddard> quite a few patches open, would be nice to get them moving
15:32:26 <mgoddard> yes
15:32:47 <mgoddard> I will go through and add RP+1 to some to focus attention
15:33:12 <mgoddard> anything else for kayobe?
15:33:43 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:34:33 <priteau> mgoddard: Nothing else from me
15:34:33 <mgoddard> ok, crunch time
15:34:51 <mgoddard> looking at https://etherpad.opendev.org/p/KollaWhiteBoard, L187
15:34:51 <hrw> ;D
15:35:30 <mgoddard> mnasiadka: contributor & PTL docs need an update, although it's not affected by feature freezr
15:35:39 <mnasiadka> mgoddard: on my list
15:35:42 <mgoddard> cool
15:36:06 <mnasiadka> mgoddard: what about docker healthchecks for kolla-ansible - are we going to push it this time, or not?
15:36:35 <hrw> mnasiadka: I thought that we merged that...
15:36:54 <mgoddard> mnasiadka: I'm up for merging it if it gets reviews
15:37:00 <mnasiadka> hrw: Kolla part - sure
15:37:05 <osmanlicilegi> not yet but we should
15:37:20 <mnasiadka> mgoddard: https://review.opendev.org/#/c/676389/ - I'll rebase it and do RP+1 - let's see if it gets reviews.
15:37:21 <patchbot> patch 676389 - kolla-ansible - Use Docker healthchecks for core services - 49 patch sets
15:37:39 <mgoddard> ok
15:37:39 <yoctozepto> what do they give us apart from being there? :-) I'm for merging because kolla part merged but really...
15:37:41 <mnasiadka> ah, and incorporate hrw's comments
15:38:01 <mnasiadka> yoctozepto: well, that's only meaningful for an operator to understand if something is wrong :)
15:38:17 <yoctozepto> mnasiadka: it's wrong when it's not worky!
15:38:27 <mnasiadka> yoctozepto: and healthcheck says "not worky!"
15:38:36 <mgoddard> I guess if you were monitoring the container state
15:38:43 <mgoddard> you could alert on it
15:38:51 <mnasiadka> yeah, another idea is to do monitoring and alerting
15:39:21 <mnasiadka> I would say let's merge it - it doesn't hurt anybody - maybe we'll find some good way to integrate it with monasca alerting or something similar
15:39:35 <mnasiadka> I mean of course after reviews :)
15:39:41 <mnasiadka> will rebase soon and ask for reviews
15:39:51 <yoctozepto> mnasiadka: yeah, I'm in
15:40:00 <yoctozepto> first ovn though, more important, today, I remember
15:40:02 <hrw> ussuri release... I want to get aarch64 CI for centos8/debian/ubuntu. which now is outside of kolla. for k-a still 2 patches for CI so can have debian/source train->ussuri checks
15:40:10 <mgoddard> just going down the list...
15:40:19 <mgoddard> [kolla ansible] performance improvements
15:40:29 <mgoddard> we merged something for fact caching
15:40:32 <mgoddard> it's a start
15:40:38 <hrw> and then just some tweaks here and there as usual
15:40:43 <yoctozepto> yay :D
15:40:51 <mgoddard> probably needs some docs on ansible tuning
15:40:52 <yoctozepto> we so performant <3
15:40:55 <mgoddard> [kolla ansible] Nova cells v2
15:41:11 <mgoddard> I can't see us getting to this, I'll mark it pushed to V
15:41:31 <yoctozepto> yeah, it's been a tough cycle :/
15:41:42 <mgoddard> [kolla ansible] TLS everywhere
15:41:47 <yoctozepto> ALMOST!
15:41:48 <yoctozepto> :D
15:41:49 <mgoddard> we merged keystone
15:42:02 <mgoddard> there are patches for heat, glance, cinder
15:42:15 <yoctozepto> I guess it's doable for the core
15:42:23 <mgoddard> generalfuzz: if you have patches for other services, would be great to see them
15:43:16 <mgoddard> [kolla ansible] More host-level commands (day 2 ops)
15:43:30 <mgoddard> Add a new compute host to /etc/hosts
15:43:51 <mgoddard> seems we no longer need this as much with the live migration fix
15:44:07 <osmanlicilegi> agree
15:44:26 <scottsol> restart would still be nice to have
15:44:41 <mgoddard> scottsol: true
15:45:13 <mgoddard> [kolla] infra images
15:45:16 <mgoddard> pushing to V
15:45:29 <mgoddard> [kolla-ansible] Capture Docker logs
15:45:37 <yoctozepto> mgoddard: unless you run tls because then you need up-to-date name resolution
15:45:45 <yoctozepto> that said, then you probably should rely on dns...
15:45:54 <mgoddard> right
15:46:02 <mgoddard> [kolla ansible] OVN support
15:46:06 <mgoddard> has 1x +2
15:46:10 <yoctozepto> OVN doing great
15:46:24 <yoctozepto> I think it will get another one, it was close to mine before
15:46:30 <mgoddard> cool
15:46:43 <mgoddard> oven ready
15:46:54 <mgoddard> [kolla ansible] Ansible lint
15:47:03 <mgoddard> seems we got quite close to this?
15:47:15 <yoctozepto> it lies somewhere
15:49:47 <mgoddard> most of the 'other things' didn't get done
15:50:19 <mgoddard> yoctozepto: any thoughts on https://review.opendev.org/#/c/699445/ ?
15:50:20 <patchbot> patch 699445 - kolla-ansible - Add Ansible group check to prechecks - 4 patch sets
15:50:26 <openstackgerrit> Merged openstack/kolla master: gnocchi: install missing python3-wheel package  https://review.opendev.org/720159
15:50:32 <mnasiadka> mgoddard: for ansible-lint I'll check the status of required changes to make it pass and ping you & yoctozepto tomorrow
15:50:37 <openstackgerrit> Merged openstack/kolla-ansible master: Fix Designate not to use etcd coordination backend  https://review.opendev.org/719583
15:50:45 <mgoddard> mnasiadka: cool
15:50:57 <mgoddard> it's not critical - not really a feature :)
15:51:02 <yoctozepto> mgoddard: was it missing only full coverage afair?
15:51:15 <yoctozepto> that precheck I mean
15:51:24 <mgoddard> yoctozepto: I picked some bad examples in my testing that it couldn't handle
15:52:02 <yoctozepto> mgoddard: oh well, but it's not making anything worse, right?
15:52:09 <mgoddard> no
15:52:29 <yoctozepto> send mnasiadka at it, and I will re-review later too
15:53:15 <mgoddard> ok
15:53:47 <mgoddard> still a couple of loose ends for CentOS 8 too, will try to tidy up soon
15:53:51 <mgoddard> #topic Virtual PTG
15:54:21 <hrw> several teams started mentioning vptg
15:54:31 <mgoddard> June 1 - 5
15:54:49 <mgoddard> I haven't read the latest email about it
15:55:32 <mgoddard> but it will be nice to do our PTG at the same time as everyone else again
15:55:42 <hrw> yep
15:55:45 <hrw> and I have proper mic now ;D
15:56:10 <osmanlicilegi> we will miss donald duck :]
15:56:19 <hrw> but also know how to switch webcam one from 'Donald Duck' to 'quite normal, sometimes in a bucket'
15:56:46 <mgoddard> I think we'll need to decide how many sessions we want
15:57:04 <hrw> https://www.amazon.com/Trust-GXT-212-Mico-Microphone/dp/B07LC43PXH works good
15:57:21 <hrw> mgoddard: kolla, kolla-ansible, kayobe, klub?
15:57:45 <mgoddard> something like that
15:57:53 <mgoddard> klub is probably separate?
15:57:55 <hrw> I can provide zoom without time limit with potential recording
15:58:06 <mgoddard> of course klub members welcome to join in
15:58:20 <mgoddard> that would be good
15:58:39 <mgoddard> although I think both previous PTGs we tried zoom then switched to meet
15:58:56 <yoctozepto> osmanlicilegi, hrw: I hardly survived last donald duck session, it was too funny ;D
15:59:04 <osmanlicilegi> meet is simple as it seems I think
15:59:04 <hrw> I use zoom 3 times per week with 3-23 people at same time.
15:59:21 <yoctozepto> mgoddard: what was the issue?
15:59:25 <yoctozepto> mgoddard: with zoom
15:59:30 <mgoddard> #action mgoddard to read PTG emails and request slots
15:59:34 <hrw> osmanlicilegi: I am out if gmeet is in use
15:59:35 <mgoddard> can't remember
15:59:51 <hrw> osmanlicilegi: it does not work on my desktop
16:00:02 <yoctozepto> well, except with hrw, gmeet works best, but it is not usable by Chinese folks
16:00:07 <mgoddard> quick final topic
16:00:10 <mgoddard> #topic Train release planning (centos 8)
16:00:16 <mgoddard> Who added it?
16:00:27 <hrw> I did
16:00:33 <yoctozepto> oh why
16:00:56 <hrw> as we had ussuri release planning in there I would like to see some train release as we have it finally working with c8
16:01:14 <yoctozepto> makes sense
16:01:51 <hrw> so we can say 'pip install kolla==9.train.version' and you are free to go
16:02:01 <hrw> instead of 'fetch head of stable/train'
16:02:25 <mgoddard> ok
16:02:33 <hrw> we have kolla stable/* in stable situation so let release stein/train
16:02:40 <mgoddard> acceptance criteria at the bottom of https://etherpad.opendev.org/p/kolla-centos8
16:03:02 <mgoddard> needs https://review.opendev.org/717807
16:03:03 <patchbot> patch 717807 - kolla (stable/train) - Add docs and release note for CentOS 8 - 1 patch set
16:03:29 <mgoddard> and need to switch from delorean to centos-release-openstack-train
16:03:34 <yoctozepto> ^ this
16:03:55 <yoctozepto> also, binary seems brokish
16:03:59 <yoctozepto> maybe due to delorean
16:04:00 <hrw> +2+W
16:04:18 <mgoddard> yeah
16:04:32 <mgoddard> anyone want to pick up delorean?
16:04:50 <hrw> can do
16:04:57 <mgoddard> thanks
16:05:16 <mgoddard> #action hrw to switch kolla centos 8 train from delorean to centos-release-openstack-train
16:05:21 <mgoddard> (if it exists)
16:05:33 <mgoddard> (if not, badger RDO)
16:05:58 <mgoddard> ok, we're over time
16:06:10 <mgoddard> thanks y'all
16:06:12 <mgoddard> #endmeeting