15:00:15 <mgoddard> #startmeeting kolla
15:00:15 <openstack> Meeting started Wed Oct  2 15:00:15 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:18 <openstack> The meeting name has been set to 'kolla'
15:00:21 <mgoddard> #topic rollcall
15:00:39 <mgoddard> (╯°□°)╯︵ ┻━┻
15:00:44 <scottsol> o\
15:00:46 <hrw> o/
15:00:50 <JamesBenson> o\
15:00:57 <yoctozepto> o/
15:02:01 <openstackgerrit> Mark Goddard proposed openstack/kolla stable/stein: swift-proxy-server: use Python 3 for Debian/Ubuntu  https://review.opendev.org/686178
15:02:53 <mgoddard> #topic agenda
15:03:05 <mgoddard> * Roll-call
15:03:08 <mgoddard> * Announcements
15:03:10 <mgoddard> ** Kolla feature freeze this week
15:03:12 <mgoddard> ** OpenStack Train RC1s available
15:03:14 <mgoddard> * Review action items from last meeting
15:03:16 <mgoddard> * CI status
15:03:18 <mgoddard> * Train release planning
15:03:20 <mgoddard> * Review priorities
15:03:22 <mgoddard> * Debian CI changes
15:03:24 <mgoddard> ** kolla-ansible Debian/source - https://review.opendev.org/686133 https://review.opendev.org/686134
15:03:26 <mgoddard> ** publishing images https://review.opendev.org/686143
15:03:28 <mgoddard> * Short-term and long-term proposal for managing non-OpenStack projects Docker images (e.g. sensu case)
15:03:30 <mgoddard> * Adding priorities to images at build time (I want 'openstack-base' to be built asap)
15:03:32 <mgoddard> #topic announcements
15:03:56 <generalfuzz> o/
15:04:14 <mgoddard> #info Kolla feature freeze this week
15:04:27 <mgoddard> #info OpenStack Train RC1s available
15:04:37 <mgoddard> Any others?
15:05:39 <mgoddard> #topic Review action items from last meeting
15:05:47 <mgoddard> egonzalez to ask about glance-store config for tacker on openstack-discuss
15:06:04 <mgoddard> I didn't see such an email
15:06:08 <mgoddard> anyone else want to pick it up?
15:06:36 <yoctozepto> pick up writing mail?
15:06:41 <mgoddard> yes
15:06:54 <yoctozepto> geez, is that so hard :D
15:07:05 <yoctozepto> pin it at me then
15:07:09 <mnasiadka> o/
15:07:15 <mgoddard> well, and following up on the response
15:07:32 <mgoddard> updating that tacker review to match
15:07:56 <mgoddard> #action yoctozepto to ask about glance-store config for tacker on openstack-discuss
15:08:01 <mgoddard> start with the mL
15:08:14 <mgoddard> #topic CI status
15:08:43 <mgoddard> #link https://etherpad.openstack.org/p/KollaWhiteBoard
15:09:24 <mgoddard> how we looking?
15:09:31 <mgoddard> only pike is broken for kolla now, right?
15:10:55 <openstackgerrit> Michal Nasiadka proposed openstack/kayobe master: Fix customised inventory section  https://review.opendev.org/686141
15:11:07 <hrw> and pike is just +2 away iirc
15:11:10 <mgoddard> NFV scenario still broken due to tacker issues
15:11:42 <hrw> just gave +W for pike/fluentd
15:11:57 <mgoddard> k
15:12:32 <mgoddard> Kayobe was broken, now fixed
15:12:36 <yoctozepto> mostly fine
15:12:45 <mgoddard> kolla-cli - ???
15:12:55 <yoctozepto> no idea, low activity
15:13:25 <openstackgerrit> Pierre Riteau proposed openstack/kayobe master: Fix customised inventory section  https://review.opendev.org/686141
15:13:26 <mgoddard> looks green on master at least
15:14:00 <mgoddard> #topic Train release planning
15:14:25 <mgoddard> As mentioned, we're planning to enter feature freeze by the end of this week
15:14:44 <mgoddard> I expect there will be some exceptions requested
15:15:16 <mgoddard> what will our policy be?
15:15:47 <mgoddard> features wanting to merge to master after EOD on Friday need to request an FFE?
15:15:48 <jovial[m]> strict, but fair :)
15:16:05 <hrw> mgoddard: +1 for it
15:16:22 <mnasiadka> I would say normal policy :) +1
15:16:23 <mgoddard> I expect I'll need one for cells
15:16:39 <hrw> mgoddard: and RP-1 for anything which is not train priority?
15:16:56 <yoctozepto> +1
15:16:59 <mgoddard> hrw: yeah
15:17:06 <jovial[m]> FFE?
15:17:06 <yoctozepto> (exception for ipv6 pretty please)
15:17:15 <mgoddard> feature freeze exception
15:18:40 <mgoddard> we have had some less active cores who don't attend meetings start adding +2 after feature freeze previously
15:19:06 <mnasiadka> punish them ;-)
15:19:09 <mgoddard> how can we avoid this?
15:19:19 <mgoddard> email to openstack-discuss?
15:19:24 <mgoddard> personal email?
15:19:29 <yoctozepto> personal I think
15:19:32 <mgoddard> k
15:19:45 <mgoddard> #action mgoddard to email cores about feature freeze
15:20:02 <mgoddard> I'll include openstack-discuss too
15:20:08 <mgoddard> BCC cores
15:20:30 <hrw> ok
15:20:35 <mnasiadka> makes sense
15:20:48 <mgoddard> we probably need to find a hard freeze date
15:20:53 <yoctozepto> make it sound nice :-)
15:21:00 <yoctozepto> end of oct?
15:21:03 <mgoddard> yoctozepto: of course :)
15:21:35 <mgoddard> yoctozepto: that's quite a long time after FF
15:21:46 <hrw> mgoddard: typical 'two weeks' then?
15:21:47 <yoctozepto> :V
15:21:56 <mgoddard> hrw: that sounds about right
15:22:15 * yoctozepto remembers 'two weeks' taking 3 months
15:22:15 <mgoddard> we can revisit next week if necessary
15:22:25 <hrw> this week is 'add whatever you have', then two weeks for 'ops, bug fixes go' and then release
15:22:26 <mgoddard> yeah, not planning on that again
15:22:53 <mgoddard> well remember we are still dependent on RDO
15:23:03 <mgoddard> and whatever surprises they throw at us :)
15:23:14 <hrw> mgoddard: what about: next meeting we review what we have in FFE/queue, next week we mark what is not going, then end
15:23:21 <mgoddard> anyone know status of RDO train repos?
15:23:32 <hrw> just started centos/binary arm64 build
15:23:33 <mgoddard> hrw: seems fair
15:24:47 <mgoddard> For anyone interested, we now have some docs on our release process: https://docs.openstack.org/kolla/latest/contributor/release-management.html
15:25:10 <yoctozepto> I read that once but will re-read
15:25:18 <mnasiadka> mgoddard: they are working on rc1, https://review.opendev.org/#/q/topic:train_release
15:25:18 <yoctozepto> I want to finish ipv6
15:25:27 <yoctozepto> and then help with kolla docs on the support matrix
15:25:30 <yoctozepto> (kolla-ansible)
15:25:41 <yoctozepto> then could help rdo
15:25:43 <mgoddard> nice
15:25:53 <mnasiadka> mgoddard: and this: https://review.rdoproject.org/r/#/q/topic:train-jobs
15:26:17 <mgoddard> #link https://review.rdoproject.org/r/#/q/topic:train-jobs
15:26:24 <mgoddard> #link https://review.opendev.org/#/q/topic:train_release
15:26:48 <openstackgerrit> Merged openstack/kayobe master: Fix container image build with multiple regexes  https://review.opendev.org/681504
15:26:49 <hrw> 36 threds on 46 cores with /var/lib/docker in tmpfs == builds which fly
15:27:11 <mgoddard> ok, let's move on, we have a few topics this time
15:27:23 <mgoddard> #topic Review priorities
15:28:49 <mgoddard> I'm still pushing on the cells patches, but will add RP+1 once they're ready
15:29:12 <mgoddard> internal TLS has another patch ready https://review.opendev.org/664517
15:29:21 <mgoddard> they're the main priorities IIUC
15:29:46 <mgoddard> Kayobe has a couple of priority reviews which need looking at
15:29:55 <mgoddard> #link https://tiny.cc/kayobe-review-dashboard2
15:30:04 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/stein: Fix container image build with multiple regexes  https://review.opendev.org/686187
15:30:36 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/rocky: Fix container image build with multiple regexes  https://review.opendev.org/686188
15:30:55 <mgoddard> As we near the end of the release, please start using RP+1 for bug fixes
15:31:17 <hrw> ok
15:31:30 <mgoddard> (following normal rules - ready for review)
15:31:52 <mgoddard> I also wondered how many bug fixes we have sitting unreviewed
15:32:04 <mgoddard> might try searching for Closes-Bug in gerrit
15:32:11 <mgoddard> #topic Debian CI changes
15:32:20 <hrw> my stuff
15:32:22 <mgoddard> kolla-ansible Debian/source - https://review.opendev.org/686133 https://review.opendev.org/686134
15:32:25 <mgoddard> publishing images https://review.opendev.org/686143
15:32:29 <mgoddard> hrw: take it away
15:32:33 <hrw> ok.
15:32:49 <hrw> so we need kolla-ansible tests for Debian images. Source one for start
15:33:16 <hrw> https://review.opendev.org/686133 should do the job (including build of images) but fails somewhere still
15:33:55 <hrw> help would be nice as I was not involved in k-a CI before
15:34:21 <hrw> opinions? questions?
15:35:24 <hrw> https://review.opendev.org/686143 is to build and publish debian/source images on normal schedule. And on release to make also debian/binary ones as at that time we should have Debian binary packages available
15:35:29 <mgoddard> hrw: thanks for starting it
15:35:47 <hrw> mgoddard: simplifies my life ;D
15:36:16 <hrw> ones 686143 gets in and we publish images then 686133 may get simplified to remove build of images
15:36:27 <hrw> s/ones/once
15:36:32 <mgoddard> right
15:36:32 <hrw> opinions? questions?
15:37:10 <mgoddard> it makes sense to add a deploy job for debian, given we have put work into getting images building
15:37:39 <mgoddard> additional CI load is a bit concerning, but not sure what we can do about that
15:37:54 <mgoddard> a single node job isn't too much
15:38:08 <mnasiadka> well, OSA even runs Ceph jobs on single node :)
15:38:15 <mgoddard> I'd definitely suggest trying a local deploy, as you commented in the review
15:38:20 <mnasiadka> (but that's rather odd, than giving any results)
15:38:30 <hrw> mgoddard: just finished. worked fine. arm64 3 nodes, no ceph
15:38:40 <mgoddard> ARM64 :)
15:38:41 <yoctozepto> mnasiadka: multinode must be tested ;-)
15:38:55 <hrw> mgoddard: for me it is easier to have arm64 boxes than x86 ones
15:39:06 <mgoddard> hrw: x86 may be different, but hopefully it's a good sign
15:39:08 <hrw> I probably can do 30+ nodes multinode
15:39:11 <mnasiadka> yoctozepto: yeah, but we should maybe have one set of multimode jobs with Ceph included
15:39:23 <mgoddard> hrw: did you try running init-runonce, and booting a VM etc?
15:39:34 <hrw> mgoddard: it is end of deploy
15:39:46 <hrw> http://paste.debian.net/1103748/ is my test script
15:40:04 <mgoddard> ok cool
15:40:18 <mgoddard> can help with CI debugging if necessary
15:40:29 <mgoddard> good opportunity to become more familiar though :)
15:40:31 <hrw> would be great
15:40:45 <mgoddard> anyone else have thoughts?
15:41:42 <mgoddard> #topic Short-term and long-term proposal for managing non-OpenStack projects Docker images (e.g. sensu case)
15:41:51 <mgoddard> btw thanks hrw
15:41:57 <hrw> yw
15:42:01 <mgoddard> mnasiadka: I think this is you
15:42:11 <mnasiadka> Yeah
15:42:25 <mnasiadka> We have long history of images - mainly monitoring related
15:42:31 <mnasiadka> that we tend to revive when builds break
15:42:44 <mnasiadka> but we don't really check if they are usable in k-a
15:43:13 <mnasiadka> sensu, prometheus, tick stack, whatever else there is
15:43:21 <mnasiadka> Do we want to continue that way?
15:43:33 <yoctozepto> no
15:43:53 <yoctozepto> we want to deprecate everything
15:43:55 <yoctozepto> <3
15:43:58 <mnasiadka> Yay!
15:44:03 <hrw> but what kind of procedure for it?
15:44:03 <mgoddard> lol
15:44:06 <yoctozepto> xD
15:44:21 <yoctozepto> build jobs in rings
15:44:31 <yoctozepto> I think 2-3 is fine
15:44:56 <yoctozepto> no sense to break core builds if just sensu broke
15:45:00 <mgoddard> it's not limited to monitoring - plenty of less commonly used openstack services are untested
15:45:15 <jovial[m]> Can they be moved out of tree if we introduce user definable roles?
15:45:30 <mnasiadka> Yeah, that's one of the options
15:45:35 <mgoddard> jovial[m]: we'd need to do the same for kolla images
15:46:06 <hrw> or we move them out of CI
15:46:15 <mnasiadka> I would rather propose to deprecate them in U cycle (images and roles) and  introduce user definable roles
15:46:32 <mnasiadka> then a user can use his own sensu/prometheus roles deploying some upstream sensu/prometheus images
15:46:58 <mnasiadka> and the burden of maintaining this is not on this limited team
15:47:22 <mgoddard> who would maintain it?
15:47:31 <yoctozepto> community
15:47:37 <hrw> yoctozepto: who?
15:47:44 <yoctozepto> others
15:47:46 <yoctozepto> not us
15:47:47 <mgoddard> it needs an owner
15:48:20 <mnasiadka> well, user definable roles needs an owner in Kolla
15:48:45 <mnasiadka> if we introduce some framework, with which you can use your own home-baked role and images
15:49:10 <mnasiadka> (probably following our role structure and variables)
15:49:16 <hrw> this, that... can we start simple? drop whatever extras from CI will make life easier
15:49:24 <hrw> whatever is not in CI == community
15:49:35 <mnasiadka> and we stop building those?
15:49:37 <mnasiadka> but keep them in tree?
15:50:00 <yoctozepto> not stop
15:50:04 <yoctozepto> but from another job
15:50:05 <hrw> iirc periodic build push what it built
15:50:07 <yoctozepto> only periodic
15:50:08 <mgoddard> what if we look at it from the perspective of the support matrix?
15:50:19 <yoctozepto> indeed mgoddard
15:50:27 <mgoddard> we have started to define what is supported, and what is community maintained
15:50:32 <yoctozepto> include my comment regarding categorization
15:50:44 <yoctozepto> and we can get there pretty quickly
15:50:45 <mgoddard> supported and tested is currently based on kolla ansible CI
15:50:48 <yoctozepto> regarding what goes where
15:51:01 <mgoddard> but we could extend to what we gate on in kolla
15:51:10 <mgoddard> I'm rephrasing yoctozepto here I realise
15:51:24 <yoctozepto> no problem
15:51:58 <mgoddard> I know it's annoying having a load of untested code, but we do see people using things we don't realise
15:52:18 <mgoddard> and it's one of the great parts of the kolla project - it has a huge range of support
15:52:38 <mgoddard> we've started to set expectations via the matrix
15:53:09 <mgoddard> it would be a shame to just discard everything that is not tested
15:53:23 <mnasiadka> well, it's fine that it is that way - but in the end we just fix things we think nobody uses :)
15:53:39 <mnasiadka> and we just fix them to make the train go to next station
15:54:09 <mnasiadka> but maybe I'm just complaining :)
15:54:16 <mgoddard> yes :p
15:54:29 <mgoddard> the thankless job of open source maintenance
15:54:34 <mnasiadka> haha
15:55:07 <yoctozepto> discard is a no-no
15:55:12 <mgoddard> we have a few ideas being tossed around here
15:55:27 <mgoddard> 1. deprecate and eventually remove some images/roles
15:55:36 <mgoddard> 2. extensible framework for images/roles
15:55:46 <yoctozepto> that "deprecate everything" was a joke though ;p
15:55:52 <mgoddard> despite my negativity, I'm +1 for both
15:56:01 <mnasiadka> yoctozepto: first we need to deprecate mariadb :)
15:56:06 <yoctozepto> exactly
15:56:09 <hrw> no
15:56:15 <yoctozepto> guys
15:56:20 <hrw> we need to remove "base" - no one is using it
15:56:21 <yoctozepto> long-term / short-term
15:56:31 <yoctozepto> let's focus on short-term for a moment
15:56:33 <mgoddard> on 1, let's not make sweeping deprecations, but analyse which images we can drop
15:56:43 <yoctozepto> I think we need to separate "other" images from "core"
15:56:45 <mgoddard> using our matrix as a guide
15:56:46 <hrw> shortterm? limit CI list
15:56:50 <mgoddard> oh yes
15:56:56 <yoctozepto> so that we can keep periodic churning
15:57:02 <hrw> whatever is not on CI == community.
15:57:06 <yoctozepto> especially now
15:57:10 <mgoddard> 3. define categories of gating + non-gating images
15:57:11 <hrw> and periodic building everything
15:57:12 <yoctozepto> when we build and publish at once
15:57:19 <yoctozepto> and lose everything due to sensu
15:57:25 <yoctozepto> sensu-client in fact
15:57:30 <yoctozepto> being unable to build
15:57:30 <yoctozepto> lol
15:57:36 <mnasiadka> mgoddard: yeah, 3. is probably best to achieve in short-term
15:57:46 <yoctozepto> define and act upon
15:58:42 <mgoddard> these are all things we could look at during Ussuri
15:58:56 <mgoddard> I think we have enough to be getting on with for Train
15:59:34 <yoctozepto> unfortunately yes
15:59:43 <mnasiadka> fortunately everything is fixed :)
15:59:43 <yoctozepto> I am afraid :-(
15:59:45 <mgoddard> do we need to start tracking potential work for U somewhere?
15:59:58 <yoctozepto> yup, I think so
16:00:02 <mnasiadka> mgoddard: let's open a ptg planning ether pad?
16:00:05 <yoctozepto> high time
16:00:21 <mgoddard> wow, positive response
16:00:55 <mnasiadka> we have features that have been postponed, some pain points identified - there is enough material to gather for ptg :)
16:01:48 <hrw> anyone going to ptg?
16:02:53 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ussuri-ptg
16:02:57 <mnasiadka> Shanghai? me not
16:03:09 <mgoddard> I'll populate it
16:03:18 <mnasiadka> Maybe let's organise PTG in Poland :)
16:03:28 <hrw> btw - do we support cinder on nfs?
16:04:20 <yoctozepto> yeah, PTG in Poland :D
16:04:50 <mnasiadka> hrw: there is that option in globals.yml...
16:04:56 <hrw> mnasiadka: D:
16:05:04 <hrw> yoctozepto: that's not stupid idea
16:05:13 <hrw> yoctozepto: can you arrange a room at uni?
16:06:35 <mgoddard> ok, we're over time
16:06:44 <mgoddard> hrw: mind if we push your last topic to next week?
16:06:56 <hrw> mgoddard: no problem
16:07:01 <mgoddard> k
16:07:06 <yoctozepto> hrw: yeah, that's doable - we could organize a little event for students too
16:07:09 <mgoddard> thanks everyone
16:07:14 <mgoddard> #endmeeting