15:02:49 <hrw> #startmeeting Kolla
15:02:50 <openstack> Meeting started Wed Nov 13 15:02:49 2019 UTC and is due to finish in 60 minutes.  The chair is hrw. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:54 <openstack> The meeting name has been set to 'kolla'
15:02:54 <hrw> #chair mgoddard
15:02:55 <openstack> Current chairs: hrw mgoddard
15:03:06 <yoctozepto> (paste agenda)
15:03:14 <hrw> * Roll-call
15:03:14 <hrw> * Announcements
15:03:14 <hrw> * Review action items from last meeting
15:03:14 <hrw> * CI status
15:03:16 <hrw> * Train release planning
15:03:19 <hrw> ** Deprecate ODL deploying support https://review.opendev.org/691749
15:03:21 <hrw> ** Kolla support matrix follow-up - known downstreams that use the image
15:03:24 <hrw> * Review priorities
15:03:27 <hrw> #topic roll-call
15:03:29 <hrw> o/
15:03:31 <yoctozepto> o/
15:03:33 <mnasiadka> o/
15:03:36 <scottsol> o/
15:03:39 <chason> o/
15:03:45 <yoctozepto> (reping mgoddard)
15:03:50 <mgoddard> o/
15:03:57 <mgoddard> who's driving this thing?
15:04:02 <yoctozepto> mgoddard: you are
15:04:06 <hrw> mgoddard: you
15:04:12 <mgoddard> sorry, had to take care of something. you have my full attention
15:04:21 <mgoddard> #topic announcements
15:04:48 <mgoddard> #info Kolla pvPTG last week was a great success!
15:04:55 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ussuri-ptg
15:05:04 <mgoddard> ^ notes from the discussions
15:05:12 <hrw> we had up to 13 people iirc
15:05:13 <mgoddard> thanks to everyone who joined in
15:05:16 <mgoddard> yep
15:05:19 <mgoddard> good turnout
15:05:42 <hrw> thanks to yoctozepto for organizing space for meeting in Białystok
15:05:44 <PrinzElvis> Hi all
15:05:46 <mgoddard> +1
15:05:50 <mgoddard> hi PrinzElvis
15:06:03 <mgoddard> we're just having the weekly meeting right now
15:06:22 <mgoddard> #info Vote on Kolla Ussuri priorities until the end of this week
15:06:25 <mgoddard> #link https://etherpad.openstack.org/p/kolla-ussuri-priorities
15:07:10 <mgoddard> These will help guide the development effort in Ussuri
15:07:16 <mgoddard> Any other announcements?
15:07:22 <hrw> 1
15:07:31 <hrw> ops
15:07:43 <mnasiadka> so now you need to announce something hrw
15:07:49 <yoctozepto> what ops
15:07:50 <yoctozepto> devops
15:07:52 <yoctozepto> sysops
15:07:56 <mgoddard> chatops
15:08:01 <yoctozepto> let's move on
15:08:11 <mgoddard> #topic Review action items from last meeting
15:08:22 <mgoddard> there were none
15:08:31 <mgoddard> #topic CI status
15:08:33 <yoctozepto> 100% completion success
15:09:01 <yoctozepto> (that was about actions, not CI obviously)
15:09:02 <mgoddard> I've been a little out of the loop for the last few days. How is CI looking?
15:09:18 <mgoddard> the whiteboard says GREEN
15:09:23 <mgoddard> #link https://etherpad.openstack.org/p/KollaWhiteBoard
15:09:24 <yoctozepto> no permabreaks observed
15:09:43 <hrw> mgoddard: quite ok. ubuntu/source/congress misbehave but it looks like locale thing to check
15:10:05 <yoctozepto> yeah, this is for back to ussuri stuff
15:10:15 <yoctozepto> we need to fix ussuri and train locale again
15:10:16 <hrw> mgoddard: things may break once we merge all 'move back to master' patches but centos builds with them
15:10:24 <yoctozepto> ubuntu does not buy en_US.UTF-8
15:10:46 <yoctozepto> will be fixed today (I guess)
15:11:01 <mgoddard> hmm, co C.UTF8 for ubuntu, en_US.UTF8 for centos?
15:11:29 <yoctozepto> I think so
15:11:35 <yoctozepto> centos8 should have C.UTF-8 too
15:11:50 <yoctozepto> so it might be train-only conditional
15:11:53 <mgoddard> ok
15:11:56 <yoctozepto> and revert on ussuri
15:12:07 <mgoddard> flip flop
15:12:07 <hrw> centos8 will not have C.UTF-8
15:12:14 <hrw> as it is not in upstream glibc yet
15:12:16 <yoctozepto> u sure?
15:12:21 <hrw> yes
15:12:23 <yoctozepto> hmm
15:12:29 <hrw> it was in some rhel bug
15:12:41 <yoctozepto> I thought it was about 7, but ok
15:12:52 <yoctozepto> will check and then decide, fix and be done
15:12:54 <yoctozepto> :D
15:13:00 <mgoddard> ok, move on
15:13:00 <hrw> en_US.UTF-8 is safest
15:13:08 <yoctozepto> hrw: as long as it is there
15:13:11 <yoctozepto> let's move on
15:13:18 <mgoddard> #topic Train release planning
15:13:22 <yoctozepto> (we can also install it everywhere, most natural env then)
15:13:43 <mgoddard> #info Kolla and Kolla Ansible RC1 released, stable/train branches exist
15:13:49 <yoctozepto> Train has come
15:13:50 <hrw> yay!
15:14:13 <mgoddard> #info Master branches are no longer frozen - we can start merging features again
15:14:23 <yoctozepto> <fireworks>
15:14:30 <hrw> mgoddard: we already started
15:14:50 <mgoddard> Has anyone been through recent fixes on master and backported to stable/train?
15:14:53 <yoctozepto> we did indeed
15:15:06 <yoctozepto> mgoddard: I think I did some randomly
15:15:10 <mgoddard> ok
15:15:12 <yoctozepto> but not thoroughly
15:15:29 <mgoddard> #action mgoddard to check stable/train backports
15:15:55 <yoctozepto> Comment 20 Carlos O'Donell 2019-08-29 13:40:44 UTC
15:15:55 <yoctozepto> The C.UTF-8 locale is already in RHEL 8.0.
15:15:59 <yoctozepto> hrw: ^
15:16:07 <hrw> o! had to miss
15:16:21 <mgoddard> otherwise, please test these RC1 releases. There will be an RC2 to follow shortly with fixes added since
15:17:04 <yoctozepto> yes, sir
15:17:25 <yoctozepto> I plan to upgrade to train my semi-production cloud
15:17:31 <mgoddard> nice
15:17:38 <yoctozepto> and let you know how miserable it becomes
15:17:50 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: Make fluentd-elasticsearch configuration more robust  https://review.opendev.org/661747
15:17:53 <mgoddard> There are a couple of subtopics for this one
15:18:04 <mgoddard> #topic Deprecate ODL deploying support https://review.opendev.org/691749
15:18:24 <mgoddard> #link https://review.opendev.org/691749
15:18:25 <yoctozepto> both mine
15:18:39 <yoctozepto> already had a little chat with mnasiadka
15:18:51 <yoctozepto> we can still deprecate I guess as it does not work (confirmed)
15:18:56 <mgoddard> care to summarise?
15:19:01 <mnasiadka> mgoddard: according to egonzalez - there's no integration with Neutron in this (or it doesn't work)
15:19:07 <mnasiadka> mgoddard: and there's no way anybody can use it
15:19:16 <yoctozepto> and we did not upgrade it
15:19:23 <yoctozepto> and nobody complained/complains
15:19:30 <mnasiadka> And Eduardo proposed himself on IRC to deprecate :)
15:19:47 <yoctozepto> yeah, thx mnasiadka, could not have summarized that better
15:19:54 <yoctozepto> mgoddard: ^^
15:20:17 <mnasiadka> as I told yoctozepto I see two options - either we just deprecate and then remove, or we send a mail to ML asking if anybody needs this, if no answer after a week, we deprecate and remove :)
15:20:31 <mgoddard> #link
15:20:34 <mgoddard> https://bugs.launchpad.net/kolla-ansible/+bugs?field.searchtext=opendaylight&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&f
15:20:36 <mgoddard> ield.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&fiel
15:20:39 <mgoddard> d.has_no_blueprints.used=&field.has_no_blueprints=on
15:20:42 <mgoddard> that is a little too long
15:20:47 <yoctozepto> maybe shorten
15:20:48 <yoctozepto> :D
15:20:48 <mgoddard> bugs referencing opendaylight
15:21:20 <yoctozepto> hmm, so we can close them quickly now :D
15:21:33 <mgoddard> just as an indicator of whether anyone uses it
15:21:49 <mnasiadka> mgoddard: all of them are from 2018
15:21:55 <mgoddard> people using ODL might not have upgraded to stein+ yet
15:22:11 <mnasiadka> ok then, but it doesn't mean we have any contributor that cares for ODL
15:22:17 <yoctozepto> https://bugs.launchpad.net/kolla-ansible/+bug/1835735
15:22:17 <openstack> Launchpad bug 1835735 in kolla-ansible "initrunonce fails with opendaylight" [Undecided,In progress] - Assigned to Eduardo Gonzalez (egonzalez90)
15:22:22 <yoctozepto> so it really fails with rocky
15:22:30 <yoctozepto> and we have that reported
15:23:13 <mgoddard> ok
15:23:27 <mgoddard> I guess it just means you need to deploy it yourself
15:23:37 <yoctozepto> odl is kind of orthogonal to openstack
15:23:47 <mnasiadka> well, there's the neutron-server bit that you need to configure
15:23:48 <mgoddard> so is much of what we deploy :)
15:24:07 <mgoddard> do we need to also deprecate the image?
15:24:17 <mnasiadka> but still you can use neutron-server-opendaylight and override the config
15:24:19 <yoctozepto> mgoddard: tripleo uses it
15:24:23 <yoctozepto> but overrides all things
15:24:26 <mnasiadka> yeah, tripleo uses it
15:24:38 <mgoddard> but not the opendaylight image?
15:24:40 <yoctozepto> mnasiadka suggested the new feature in ussuri
15:24:43 <yoctozepto> could solve the problem
15:24:55 <yoctozepto> where we support external image recipes
15:24:59 <mnasiadka> mgoddard: they use both, opendaylight image and the neutron-server-opendaylight
15:25:05 <yoctozepto> or the plan minimum - stop building
15:25:07 <mnasiadka> mgoddard: override a lot, as always - but still use it
15:25:23 <yoctozepto> yeah, removing would break their jobs
15:25:28 <mgoddard> override to upgrade to a newer version?
15:25:38 <yoctozepto> yup, change repo, change packages
15:25:41 <yoctozepto> newer version
15:25:48 <yoctozepto> some internal orchestrations
15:25:57 <mgoddard> ok
15:26:43 <mgoddard> #agreed Deprecate ODL
15:26:49 <mgoddard> #undo
15:26:50 <openstack> Removing item from minutes: #agreed Deprecate ODL
15:26:53 <mgoddard> #agreed Deprecate ODL in kolla-ansible
15:27:09 <yoctozepto> mhm
15:27:14 <mgoddard> Let's move on
15:27:15 <mnasiadka> in Train or in Ussuri? :)
15:27:20 <yoctozepto> in Train
15:27:28 <mgoddard> #topic Kolla support matrix follow-up - known downstreams that use the image
15:27:43 <yoctozepto> so I got this idea
15:27:51 <yoctozepto> to include in some places
15:27:56 <yoctozepto> some place*
15:28:02 <mnasiadka> docker label
15:28:04 <yoctozepto> a table similar to support matrix
15:28:13 <yoctozepto> three columns
15:28:13 <mnasiadka> and build a table based on the labels
15:28:18 <yoctozepto> image | k-a | ooo
15:28:45 <yoctozepto> mnasiadka: this info is weird to have in labels but that could be next step
15:28:56 <yoctozepto> for now I would create it
15:29:01 <yoctozepto> you would review it
15:29:05 <yoctozepto> plus some tripleo guy
15:29:12 <yoctozepto> and we would have a better outlook
15:29:27 <mgoddard> what's the goal?
15:29:32 <yoctozepto> wdyt
15:29:50 <yoctozepto> to know who is affected by each image
15:30:36 <yoctozepto> e.g. odl would be now tripleo only
15:30:41 <yoctozepto> just like ovn is atm
15:31:02 <mgoddard> sounds like a reasonable idea
15:31:14 <yoctozepto> and many more k-a only
15:31:16 <mgoddard> like all these things, it could easily go stale
15:31:25 <yoctozepto> ok, i'll propose something later this week
15:31:27 <mgoddard> particularly on the tripleo side
15:31:33 <yoctozepto> that's the problem
15:31:48 <mgoddard> but hopefully changes less than the support matrix
15:31:49 <yoctozepto> still better than searching each time
15:32:02 <mgoddard> alternative is to get consumers to document it, then link to them
15:32:22 <yoctozepto> mhm
15:32:46 <mnasiadka> Maybe let’s invite cloudnull for a meeting and discuss?
15:33:08 <mgoddard> +1
15:33:18 <mgoddard> we should get him more involved
15:33:26 <openstackgerrit> zhulingjie proposed openstack/kolla-cli master: "Switch to official Ussuri jobs"  https://review.opendev.org/694097
15:33:33 <openstackgerrit> Radosław Piliszek proposed openstack/kolla master: Revert "base: use en_US.UTF-8 locale on all distros"  https://review.opendev.org/694098
15:33:40 <mgoddard> they need to buy into the idea, otherwise it will go stale
15:33:59 <yoctozepto> +1
15:34:07 <openstackgerrit> zhulingjie proposed openstack/kolla-cli master: Switch to official Ussuri jobs  https://review.opendev.org/694097
15:34:38 <mgoddard> yoctozepto: can you get in touch with cloudnull and find a time to discuss, e.g. next week's meeting?
15:34:58 <mgoddard> he's probably still catching up after the summit
15:35:54 <yoctozepto> mgoddard: ok, give me this action, please
15:36:19 <mgoddard> #action yoctozepto: to get in touch with cloudnull and find a time to discuss kolla image consumers, e.g. next week's meeting
15:36:32 <mgoddard> #topic Review priorities
15:36:58 <mgoddard> Priority for train is still bugs
15:37:21 <mgoddard> For master, let's focus on getting it back into development mode
15:37:27 <mgoddard> And CentOS 8
15:37:38 <yoctozepto> right-yo
15:37:39 <mgoddard> Feel free to RP+1 those patches
15:38:16 <mgoddard> Creative solutions to the py2 drop breaks images problem welcome
15:38:33 <mgoddard> In fact, let's discuss it now
15:38:51 <mgoddard> #topic Handling py2 drop
15:39:12 <mgoddard> In 1 month, all projects are supposed to drop py2 support
15:39:26 <mgoddard> which means our images may fail to build
15:39:31 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla master: base: generate en_US.UTF-8 locale data for Debian/Ubuntu systems  https://review.opendev.org/694099
15:39:31 <mgoddard> (on CentOS 7)
15:39:39 <mgoddard> we've already hit it with searchlight
15:40:07 <mgoddard> what options do we have to keep ourselves going, considering CentOS 8 support still looks a way off and is out of our control
15:40:15 <hrw> I think that for c7 this is how we should handle
15:41:07 <mgoddard> ?
15:41:47 <yoctozepto> this that this hrw?
15:42:04 <mgoddard> here's one option: pin to stable/train
15:42:05 <hrw> if image fails for centos7/source then add to UNBUILDABLE and go on
15:42:31 <mgoddard> that works until we hit an image that kolla-ansible tests need
15:42:35 <hrw> once moved to c8 we revert and go
15:42:44 <hrw> mgoddard: then we mark c7 as non-voting
15:43:16 <hrw> we have to realize that c7 will just go worse and worse now
15:43:17 <yoctozepto> xD
15:43:22 <mgoddard> how about this option
15:43:37 <mgoddard> add two new images, base-centos8, openstack-base-centos8
15:43:50 <mgoddard> these include as much as possible that builds on c8
15:44:04 <hrw> mgoddard: not doable
15:44:09 <yoctozepto> feels clunky
15:44:11 <mgoddard> then we switch the base over to these when they build
15:44:16 <yoctozepto> hrw: why?
15:44:18 <hrw> mgoddard: want 12MB of centos8 build logs to check?
15:44:22 <mgoddard> it is clunky. the whole problem is clunky
15:44:31 <hrw> FAR FAR too many packages are missing
15:44:36 <mnasiadka> Centos builds as non voting :)
15:44:44 <hrw> unless you know where to look for (u)wsgi etc
15:45:32 <mnasiadka> We do it once and revert when ready
15:45:44 <hrw> no ceph, no several python3 packages etc
15:45:50 <mgoddard> I guess it might be hard to have distro_python_version take two values
15:46:53 <yoctozepto> non-voting then
15:46:57 <yoctozepto> what can we do
15:47:08 <hrw> mgoddard: https://review.opendev.org/692368 has a list
15:47:21 <yoctozepto> for image fixes we need in oldies cherry pick to train to test
15:47:22 <mgoddard> UNBUILDABLE is better IMO
15:47:41 <mgoddard> and non-voting in k-a when the time comes
15:47:42 <yoctozepto> at some point we will have almost everything breaking
15:48:05 <mgoddard> :(
15:48:11 <hrw> mgoddard: kolla-toolbox is UNBUILDABLE :D
15:48:37 <yoctozepto> :D
15:49:29 <mgoddard> ok. Let's keep CI green for as long as possible. Once it goes red we lose the ability to see genuine failures
15:49:42 <hrw> yep
15:49:58 <openstackgerrit> Merged openstack/kayobe master: Fix kolla ansible virtualenv creation on the seed  https://review.opendev.org/693030
15:50:09 <mgoddard> #topic Open discussion
15:50:20 <mgoddard> Anyone have anything today?
15:50:34 <yoctozepto> nope
15:50:37 <hrw> nope
15:52:00 <dougsz> Any thoughts on this one? https://review.opendev.org/#/c/684799/
15:52:15 <yoctozepto> not looking, but definitely deprecate
15:52:18 <yoctozepto> :D
15:52:53 <yoctozepto> ah, this is this
15:53:02 <yoctozepto> so rmq 3.8 should solve this already
15:53:03 <scottsol> I was planning on chasing that up actually
15:53:16 <yoctozepto> if mgoddard accepts that for train, I would not mind
15:53:17 <hrw> ending in 3
15:53:18 <dougsz> Have we upgraded RabbitMQ though
15:53:20 <hrw> 2
15:53:25 <hrw> 1
15:53:29 <yoctozepto> hrw: open discussion
15:53:35 <hrw> oh
15:53:40 <yoctozepto> that would be unkind
15:54:04 <hrw> sorry
15:54:07 <yoctozepto> dougsz: nope, planned for ussuri
15:54:11 <mgoddard> dougsz: we haven't upgraded RMQ. It was a discussion topic at the PTG raised by kplant, mostly to get the prometheus exporter
15:54:13 <yoctozepto> which will make this change irrelevant
15:54:28 <yoctozepto> so it's more to be in train
15:54:39 <yoctozepto> but kind of late
15:54:59 <openstackgerrit> Marcin Juszkiewicz proposed openstack/kolla master: Ubuntu: use local APT mirrors if possible  https://review.opendev.org/694104
15:55:11 <mgoddard> RMQ upgrade has one vote so far
15:55:17 <hrw> rabbitmq 3.8 requires erlang upgrade
15:55:45 <yoctozepto> it's easy if we forget other arches for the moment
15:55:51 <yoctozepto> and wait for hrw to build erlang for them
15:55:54 <hrw> for centos/x86-64 there is repo. for debian/ubuntu/x86-64 iirc too. I can provide one for aarch64 probably
15:56:26 <yoctozepto> it was not upgrade in train because it was just too late
15:56:37 <yoctozepto> when the issue was really discovered and described
15:57:49 <mgoddard> does this help dougsz or scottsol?
15:58:07 <yoctozepto> immediately
15:58:31 <yoctozepto> 3.8 has the prometheus exporter they so desire
15:58:51 <mgoddard> Any other topics?
15:59:26 <mgoddard> Out of time
15:59:30 <yoctozepto> oot
15:59:31 <mgoddard> Thanks y'all
15:59:33 <mgoddard> #endmeeting