15:01:25 <mgoddard> #startmeeting kolla
15:01:28 <yoctozepto> let's roll, it's lots to discuss today
15:01:28 <openstack> Meeting started Wed Oct  9 15:01:25 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:31 <mgoddard> #topic rollcall
15:01:33 <openstack> The meeting name has been set to 'kolla'
15:01:33 <hrw> o/
15:01:33 <mgoddard> \o
15:01:36 <yoctozepto> o/
15:01:44 <Wasaac> o/
15:01:45 <JamesBenson> o/
15:01:47 <yoctozepto> mnasiadka
15:01:53 <yoctozepto> generalfuzz
15:02:01 <jovial[m]> 0/
15:02:24 <mgoddard> jovial[m] has calendar?
15:02:24 <generalfuzz> o/
15:02:41 <mgoddard> #topic agenda
15:02:42 <jovial[m]> yep, only 2 minutes late today :)
15:02:45 <mgoddard> * Roll-call
15:02:47 <mgoddard> * Announcements
15:02:49 <mgoddard> ** Kolla in feature freeze
15:02:51 <mgoddard> * Review action items from last meeting
15:02:53 <mgoddard> * CI status
15:02:55 <mgoddard> * Train release planning
15:02:57 <mgoddard> * Review priorities
15:02:59 <mgoddard> * (hrw) Adding priorities to images at build time (I want 'openstack-base' to be built asap)
15:03:01 <mgoddard> * Reducing load of build & publishing jobs on CI & Dockerhub https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:03:03 <mgoddard> * (yoctozepto) policy regarding development in deployed projects, e.g. ironic has now split ironic-inspector into two services and uses tooz coordination - should we support this new mode of deployment? (I think we should, for that we are trailing cycles)
15:03:05 <mgoddard> * Which images should we mark as maintained in the support matrix?
15:03:07 <mgoddard> #topic announcements
15:03:09 <mgoddard> #info Kolla in feature freeze
15:03:26 <mgoddard> #info OpenStack Train final releases next week
15:03:41 <mgoddard> Any others?
15:03:59 <yoctozepto> ipv6 passed all current CI scenarios
15:04:16 <yoctozepto> (ubuntu ones)
15:04:31 <yoctozepto> (shameless plug I know)
15:04:45 <mgoddard> nice work
15:04:52 <mgoddard> #topic Review action items from last meeting
15:05:11 <mgoddard> yoctozepto to ask about glance-store config for tacker on openstack-discuss
15:05:13 <mgoddard> mgoddard to email cores about feature freeze
15:05:16 <mgoddard> I did mine
15:05:18 <mgoddard> yoctozepto?
15:05:41 <yoctozepto> I did not because egonzalez made me think he had this under control that same day :-)
15:05:46 <yoctozepto> not so sure about this now
15:06:29 <yoctozepto> will do right away after meeting, I wonder why they are releasing it in this state
15:06:31 <mgoddard> I think we found that only filestore is supported
15:06:42 <yoctozepto> but it's filestore inside tacker
15:06:46 <mgoddard> right
15:06:52 <yoctozepto> so it's tackerstore
15:06:58 <yoctozepto> your new glance backend
15:07:18 <mgoddard> I think glance-store is just a python API for storing images without hitting glance API
15:07:41 <yoctozepto> yeah, they are using this little thing directly to do filestore
15:07:44 <yoctozepto> on the local node
15:08:10 <yoctozepto> looks pretty b0rken if you ask me, but let's not slow down the meeting
15:08:14 <mgoddard> ok, let's not get side tracked
15:08:40 <mgoddard> #action yoctozepto to investigate tacker + glance store
15:08:46 <mgoddard> #topic CI status
15:08:55 <ai_ja_nai> mgoddard many thanks
15:09:00 <mgoddard> https://etherpad.openstack.org/p/KollaWhiteBoard
15:09:31 <mgoddard> I think CI is looking ok mostly
15:09:48 <yoctozepto> yeah, no current breakages
15:10:02 <mgoddard> There are a few unreliable tests, which we should look at as we get closer to release
15:10:31 <mgoddard> Kayobe CI I thought I had fixed, but does seem to fail sometimes still
15:10:59 <yoctozepto> ;-(
15:11:19 <mgoddard> it's not too bad at the moment, something to look at another day
15:11:23 <mgoddard> #topic Train release planning
15:11:30 <mgoddard> The train is approaching
15:11:47 <yoctozepto> I like trains
15:11:53 <mgoddard> we're currently in feature freeze - please don't approve feature patches unless they have been granted a feature freeze extension
15:12:00 <yoctozepto> all you had to was to catch the * train, CJ
15:12:01 <mgoddard> currently this includes IPv6 and nova cells
15:12:08 <mgoddard> (as they are priorities)
15:12:31 <yoctozepto> mgoddard: tls frozen?
15:12:48 <mgoddard> yoctozepto: TLS merged
15:13:02 <yoctozepto> mgoddard: what about unmerged ones? :D
15:13:23 <mgoddard> there is the insecure patch for CI testing
15:13:42 <mgoddard> and vmixor's CA patch which is quite new
15:14:11 <generalfuzz> I've started putting attention on to vmixor's patches
15:14:31 <yoctozepto> both need some work I think, but generalfuzz is putting working in it
15:14:51 <mgoddard> would consider an FFE request if one was made
15:15:06 <mgoddard> deadline for getting these things merged is next friday
15:15:06 <generalfuzz> I was under the the impression that we were abandoning the insecure ones
15:15:31 <yoctozepto> generalfuzz: yeah, I would prefer the vmixor's kinda approach
15:15:57 <yoctozepto> I think TLS is really close with those internal trusts
15:16:07 <mgoddard> I hadn't heard a decision on it, but it does seem like it might be the sensible option
15:16:08 <yoctozepto> just cannot help guys due to ipv6 on me
15:16:37 <yoctozepto> (and work, and life, and phd)
15:17:00 <generalfuzz> We'll see if it's done by next Friday. If not, It'll go into the next release
15:17:00 <mgoddard> ultimately we have internal TLS support in train already
15:17:07 <yoctozepto> (glad I did not forget life)
15:17:13 <mgoddard> you just need to build your own images if you have a private CA
15:17:39 <yoctozepto> which is very likely for internal though ;p
15:17:46 <mgoddard> custom CA is a nice to have IMO (we weren't even planning to do it until quite recently)
15:18:07 <yoctozepto> I think k-a should handle this stuff
15:18:23 <mgoddard> I'm not disagreeing, just saying it can be made to work without
15:18:25 <yoctozepto> so maybe let's split this onto Ussuri if we cannot handle it
15:18:35 <yoctozepto> indeed it's doable
15:18:55 <mnasiadka> yes, it's the time of the release planning to chop off stuff we can't do :)
15:19:38 <mgoddard> I imagine generalfuzz will be working on it now regardless of which release it goes into, right?
15:19:45 <yoctozepto> we should market it as moving internal tls from pita to pitb
15:20:40 <mnasiadka> mgoddard: as long as it's not breaking anything - we can back port it later
15:21:02 <mgoddard> mnasiadka: well not really according to stable policy
15:21:24 <yoctozepto> too much of a feature probably
15:21:31 <mnasiadka> mgoddard: I don't think kolla was always following stable-policy in 100% :)
15:21:57 <mgoddard> let's see how the code develops. If it turns out to be ready to go before next week it could be given an FFE
15:22:23 <mnasiadka> yup
15:22:37 <mgoddard> I'll start working through the release process soon
15:22:44 <mgoddard> #link https://docs.openstack.org/kolla/latest/contributor/release-management.html
15:23:08 <mgoddard> if anyone wants to follow along let me know and I can keep you in the loop of what's happening
15:23:25 <mgoddard> #topic Review priorities
15:23:35 <mnasiadka> if you need any help - just shout, maybe we should have an ether pad for that :)
15:23:39 <mgoddard> thanks
15:23:49 <yoctozepto> mgoddard: count me in
15:24:03 <mgoddard> I think we know our main priorities - cells and IPV6
15:24:40 <mgoddard> I have one issue to fix before cells is ready to merge, but it's ready for reviews
15:24:48 <yoctozepto> yay
15:24:59 <yoctozepto> and that one is ubuntu binary?
15:25:00 <mgoddard> how is IPv6 looking?
15:25:29 <yoctozepto> mgoddard: as mentioned, all scenarios are passing, needs applying left comments and creating unit tests
15:25:37 <mgoddard> no, binary fixed/fudged by letting nova-api see the cell DB
15:25:52 <yoctozepto> ah
15:26:16 <mgoddard> issue is ordering of DB sync in the API and cells - need to DB sync cells before starting global services
15:26:34 <yoctozepto> I see
15:26:43 <mgoddard> is IPv6 ready for reviews?
15:27:23 <yoctozepto> mgoddard: yes, you can do second pass, but I did not apply all the comments before so it's going to get -1 from you anyway :-)
15:27:35 <mgoddard> all: if you have bug fix patches open against master that you think shold make the release, please mark RP+1
15:28:28 <mnasiadka> mgoddard: btw, in governance repo we are missing stable:follows-policy :-)
15:28:54 <mgoddard> are you saying we should add it?
15:29:08 <mgoddard> might come with some baggage
15:29:24 <mgoddard> do other deployment tools have it?
15:29:43 <mgoddard> something for open discussion, if we make it
15:29:47 <mgoddard> #topic (hrw) Adding priorities to images at build time (I want 'openstack-base' to be built asap)
15:29:53 <mgoddard> hrw: you're up
15:30:05 <mnasiadka> mgoddard: nope, neither tripleo, nor OSA
15:30:17 <hrw> https://review.opendev.org/686587 implements that
15:30:32 <mnasiadka> mgoddard: just saying that following stable policy is not something we sworn - it makes sense to follow it though :)
15:30:38 <hrw> my tests with 46 threads (in tmpfs) did not show any improvements
15:30:48 <yoctozepto> mnasiadka, mnasiadka: we can be flexible
15:31:25 <mgoddard> hrw: how many times did you test? I guess it was random
15:31:28 <hrw> planning to do some 8 threads runs to check does it make a difference
15:31:41 <hrw> mgoddard: several. centos/debian binary/source
15:32:06 <hrw> with build times in ~1 minute difference (of 35)
15:32:21 <mgoddard> fewer threads sounds sensible
15:32:42 <hrw> will do some runs on slower hw too
15:32:51 <hrw> have old 8core system
15:33:28 <mgoddard> ok anything more to discuss here? It was an old topic, before the patch
15:33:51 <yoctozepto> I think it probably builds reasonably well to not affect queue saturation
15:34:00 <yoctozepto> let's move on
15:34:03 <yoctozepto> hrw: ok?
15:34:06 <hrw> go
15:34:22 <mgoddard> #topic Reducing load of build & publishing jobs on CI & Dockerhub https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:34:31 <mgoddard> #link https://etherpad.openstack.org/p/kolla-train-image-evaluation
15:34:43 <hrw> the old topic...
15:34:56 <mgoddard> this one again
15:35:03 <mgoddard> kolla images are large
15:35:20 <mgoddard> we have many of them, and many variations
15:35:26 <hrw> and docker squash support is a joke
15:35:31 <yoctozepto> xD
15:35:40 <mgoddard> hrw: expand?
15:35:45 <mnasiadka> so, what is the biggest strain on CI? images size or time to push them to docker hub?
15:35:48 <mgoddard> (desquash?)
15:36:10 <mgoddard> mnasiadka: bandwidth I think
15:36:22 <hrw> mgoddard: either use docker-squash python and eat lot of i/o or enable experimental squash support in docker itself and look how it explodes
15:36:40 <mgoddard> hrw: ok
15:36:47 <mnasiadka> mgoddard: well, if only we could have some docker hub proxy in each of the node pool providers - we could push fast and then it can run in the background
15:37:15 <mnasiadka> I don't think there's any solution for that, we won't reduce size by 50%
15:37:23 <mgoddard> it's probably more the total bytes sent that is the problem
15:37:57 <mgoddard> IMO our best bet is to just trim down the number of images that we push
15:38:05 <mgoddard> there are a few ways to do that
15:38:12 <mnasiadka> or don't push every day :-)
15:38:18 <mgoddard> EOL pike is an easy one
15:38:23 <hrw> @weekly would be sane imho
15:38:38 <mgoddard> well that's fine until something breaks
15:38:42 <mnasiadka> hrw: as long as we really push, not fail :)
15:39:02 <mgoddard> then we could be stuck with duff images for a week
15:39:05 <mnasiadka> mgoddard: we would need to have access to trigger a job
15:39:17 <mgoddard> I was thinking of using our TBD categories
15:39:25 <mgoddard> core images go every day, other things less often
15:39:48 <mgoddard> we have a few ideas in the pad
15:40:03 <mgoddard> EOL pike - does anyone disagree?
15:40:07 <mnasiadka> or maybe there's a node pool provider that doesn't really complain about the bandwidth used - and we can stick to that?
15:40:11 <mnasiadka> +1 for EOL pike
15:40:18 <mgoddard> or at least disable publishing
15:40:45 <yoctozepto> +1 eol pike
15:41:24 <mnasiadka> I have a question by the way - do we clean up old images in docker hub, or just keep the whole history of OpenStack in there? :)
15:42:00 <mgoddard> #agreed EOL pike
15:42:09 <hrw> +1
15:42:10 <mgoddard> #action mgoddard to EOL pike
15:42:28 <mgoddard> that's 16GB x 6 per day saved
15:42:42 <mgoddard> mnasiadka: we don't clean up old images AFAIK
15:42:47 <hrw> mnasiadka: as someone who had to take care of old cloud I would say: leave them
15:43:02 <mnasiadka> fine by me - just out of curiosity :)
15:43:14 <hrw> if you deployed ocata with use of k-a and you got extra rack of hw then you can still deploy ocata to it
15:43:27 <hrw> despite ocata being EOL
15:43:53 <mgoddard> hrw: your publishing timeout patch, does it need backporting?
15:44:02 <mnasiadka> ocata is EOL 9 months, I was talking about 3 year old images :)
15:44:09 <hrw> mnasiadka: same rule
15:44:17 <hrw> mgoddard: do not think so
15:44:37 <mgoddard> hrw: why not?
15:44:51 <mnasiadka> it would be good to keep the same timeout values in all supported branches
15:45:06 <mgoddard> we do see timeouts on our stable publishing jobs
15:45:16 <hrw> mgoddard: then backport. will +2
15:46:00 <mgoddard> #action mgoddard to backport publishing job timeout increase
15:46:26 <mgoddard> Now for the big one
15:46:27 <mgoddard> We build and publish some images which are not used.
15:46:30 <mgoddard> We have deprecated many, but should we just be bold and remove them in Train and earlier?
15:46:52 <hrw> remove in train
15:46:54 <yoctozepto> I don't complain
15:47:09 <hrw> backporint to previous would be removing features which may be against policy
15:47:09 <mgoddard> https://docs.openstack.org/releasenotes/kolla/unreleased.html#deprecation-notes
15:47:49 <mnasiadka> mgoddard: what about just not publishing the deprecated ones?
15:47:51 <mgoddard> almanach, dind, kolls-k8s stuff, dragonflow
15:48:08 <mnasiadka> and maybe not building them in CI
15:48:12 <mgoddard> mnasiadka: that's another option.
15:48:13 <mnasiadka> leave the code there, we remove them in U
15:48:29 <mnasiadka> maybe even move to a ,,deprecated'' subdir
15:48:45 <mgoddard> how would we skip them?
15:48:53 <hrw> mgoddard: remove from profiles
15:49:01 <mnasiadka> yup
15:49:08 <mgoddard> do we use profiles in CI?
15:49:35 <mgoddard> I don't think so
15:49:43 <mgoddard> just build every supported image
15:49:48 <yoctozepto> mgoddard: what profiles
15:49:51 <yoctozepto> in k-a we use them
15:49:54 <yoctozepto> per scenario
15:49:58 <mgoddard> nope
15:50:06 <mnasiadka> yeah, we build them all probably - it's not hard to add couple of lines of code to skip the deprecated/* ones
15:50:18 <mgoddard> ok maybe we do in k-a
15:50:22 <yoctozepto> xD
15:50:43 <yoctozepto> kolla profiles come with this categorization I proposed
15:51:54 <mgoddard> I'm not sure how we'd do this
15:52:06 <mgoddard> and we need to do it quickly if we're going to do it for train
15:52:12 <mgoddard> anyone want to pick it up
15:52:19 <mgoddard> ?
15:52:54 <mnasiadka> I can give it a quick go tomorrow to see if we can do it fast :-)
15:53:10 <mgoddard> ok
15:53:28 <mgoddard> #action mnasiadka to look at not building or publishing deprecated images
15:53:29 <mgoddard> thanks
15:53:36 <yoctozepto> mnasiadka: use regexp :-)
15:53:44 <mnasiadka> yoctozepto: use the force Luke? :)
15:53:59 <yoctozepto> no, --force is bad
15:54:04 <mgoddard> let's revisit the other things another time
15:54:14 <mgoddard> we have some easy wins
15:54:26 <mgoddard> #topic (yoctozepto) policy regarding development in deployed projects, e.g. ironic has now split ironic-inspector into two services and uses tooz coordination - should we support this new mode of deployment? (I think we should, for that we are trailing cycles)
15:54:35 <mgoddard> long topic
15:54:39 <mgoddard> yoctozepto: it's yours
15:55:59 <yoctozepto> yeah, mine
15:56:14 <yoctozepto> I explicitly added all the details in there in case I was not around
15:56:22 <yoctozepto> should we / should not we? :D
15:56:48 <yoctozepto> we are trailing for a reason
15:57:06 <mgoddard> you mean, should we support features added in the same cycle?
15:57:14 <yoctozepto> indeed ;D
15:57:24 <yoctozepto> (since we are deploying THAT cycle)
15:58:04 <mgoddard> it's not something we've ever prevented, given enough time to develop & test on our side
15:58:22 <mgoddard> as always, just needs people to develop, test & review the code :)
15:58:23 <yoctozepto> sure but should not that be our goal
15:58:31 <yoctozepto> could come with categorization...
15:58:37 <yoctozepto> (or M/T status)
15:59:09 <mnasiadka> should be our goal, see an army of developers tracking every project changes and implementing them in k-a? :)
15:59:34 <mgoddard> we're maintainers - our goal is to keep the project maintained & healthy
16:00:00 <mgoddard> developing features can be done by maintainers, but you're wearing a different hat
16:00:24 <mgoddard> we can use priorities & design discussions to try to drive the direction of development
16:00:43 <mgoddard> but at the end of the day, people just push what they want at you :)
16:00:55 <yoctozepto> I see, got ya
16:01:27 <mgoddard> if you have time to work on that kind of thing, we'll happily accept it
16:01:54 <mgoddard> that's just my view of course :)
16:02:23 <mnasiadka> yeah, maintaining the project is prio:1, we all add some features in the background
16:02:27 <mgoddard> we can set goals, but don't always have the manpower to carry them out
16:02:59 <mgoddard> we're out of time unfortunately
16:03:08 <mgoddard> we'll have to discuss "Which images should we mark as maintained in the support matrix?" next time
16:03:28 <mgoddard> let's aim to get the support matrix merged without any Ms
16:03:35 <yoctozepto> +1
16:03:42 <yoctozepto> let's call it a meeting
16:03:58 <mgoddard> thanks all, good discussion today
16:04:06 <mgoddard> #endmeeting