15:01:28 <mgoddard> #startmeeting kolla
15:01:29 <openstack> Meeting started Wed Nov 27 15:01:28 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:32 <openstack> The meeting name has been set to 'kolla'
15:01:33 <mgoddard> #topic rollcall
15:01:36 <mgoddard> \o
15:01:41 <yoctozepto> o/
15:01:45 <hrw> o/
15:01:49 <scottsol> o/
15:04:19 <mgoddard> #topic agenda
15:04:27 <mgoddard> * Roll-call
15:04:29 <mgoddard> * Announcements
15:04:31 <mgoddard> ** Ussuri priorities now listed on whiteboard https://etherpad.openstack.org/p/KollaWhiteBoard
15:04:33 <mgoddard> ** PTG summary on openstack-discuss: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011084.html
15:04:35 <mgoddard> ** Infra images discussion on openstack-discuss: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011120.html
15:04:37 <mgoddard> ** CentOS 8 host migration discussion on openstack-discuss: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011130.html
15:04:39 <mgoddard> * Review action items from last meeting
15:04:41 <mgoddard> * CI status
15:04:43 <mgoddard> * Train release planning
15:04:45 <mgoddard> * Ussuri release planning
15:04:47 <mgoddard> * Review priorities
15:04:49 <mgoddard> ** Ceph orchestrator
15:04:51 <mgoddard> #topic announcements
15:04:53 <mgoddard> #info Ussuri priorities now listed on whiteboard
15:04:55 <mgoddard> #link https://etherpad.openstack.org/p/KollaWhiteBoard
15:05:05 <mgoddard> #info PTG summary on openstack-discuss
15:05:11 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011084.html
15:05:17 <mgoddard> #info Infra images discussion on openstack-discuss
15:05:22 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011120.html
15:05:29 <mgoddard> #info CentOS 8 host migration discussion on openstack-discuss
15:05:35 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011130.html
15:06:02 <mgoddard> Please join in these discussions as we try to firm up some of the more squishy features
15:06:07 <mgoddard> Any others?
15:06:49 <mgoddard> #topic Review action items from last meeting
15:07:05 <mgoddard> mgoddard to tidy up ussuri priorities and add to whiteboard
15:07:07 <mgoddard> check
15:07:13 <mgoddard> #topic CI status
15:07:24 <mgoddard> "kolla-toolbox fails to build on Ubuntu due to libssl older than libssl-dev"
15:07:34 <mgoddard> that one is 'fixed' now right?
15:07:46 <hrw> CI passed through it
15:08:42 <mgoddard> ok, reverted whiteboard to green
15:08:50 <mgoddard> #topic Train release planning
15:09:00 <mgoddard> I think we're getting close
15:09:17 <mgoddard> https://launchpad.net/kolla/+milestone/9.0.0
15:09:19 <yoctozepto> pending is the rmq issue
15:09:36 <yoctozepto> osmanlicilegi to work on that but not finished yet
15:09:37 <mgoddard> yeah
15:09:43 <mgoddard> is it release blocking?
15:09:52 <hrw> I would release with 3.7.10
15:09:55 <yoctozepto> other one is https://review.opendev.org/695804
15:09:56 <patchbot> patch 695804 - kolla-ansible - [Train] Fix Tacker config for the local Glance store - 3 patch sets
15:10:03 <yoctozepto> just waiting for you
15:10:16 <yoctozepto> mgoddard, hrw: if we don't mind mid-cycle upgrade
15:10:33 <yoctozepto> could work, not breaking completely obviously
15:10:37 <hrw> yoctozepto: at least it would be tested in ussuri.
15:10:38 <mgoddard> yoctozepto: I +2 tacker
15:10:54 <yoctozepto> mgoddard: I know, and send my thanks
15:10:56 <hrw> yoctozepto: now it is 'we have train built/tested, lets add fresh rmq and release'
15:11:22 <mgoddard> hrw: we're not talking about an rmq upgrade, "only" erlang
15:11:29 <hrw> yoctozepto: we completely lack !x86 support for rmq/erlang
15:11:29 <yoctozepto> hrw: I don't mind too much
15:11:38 <mgoddard> eitehr way it's still risky
15:11:47 <yoctozepto> mgoddard: yeah, only erlang and rmq is patch level
15:12:05 <yoctozepto> yeah, because it is erlang dribing most of networking there
15:12:09 <yoctozepto> driving*
15:14:22 <mgoddard> hmm, actually it is a rmq patch upgrade, missed that
15:14:43 <yoctozepto> mgoddard: no point in leaving bugs since they were blocked by erlang
15:14:50 <yoctozepto> though still it's erlang more worrying
15:15:01 <yoctozepto> the plus fact is that it's a configuration "tested" by rmq upstream
15:15:18 <yoctozepto> so we are possibly getting them to consider our bug reports as worthy ;p
15:16:10 <mgoddard> scottsol: I think this is your issue: https://bugs.launchpad.net/kolla/+bug/1814233
15:16:10 <openstack> Launchpad bug 1814233 in kolla "rabbitmq image fails to build" [Critical,Fix released]
15:16:46 <scottsol> yeh still stuck with that
15:17:07 <scottsol> erlang-hipe and the pinning of rmq wont work
15:17:20 <yoctozepto> well, not really stuck, it remains to patch tripleo and provide aarch64 repos to kolla
15:17:51 <mgoddard> how close are we to aarch64 repos?
15:18:15 <yoctozepto> I think hrw has finished them
15:18:19 <hrw> not quite
15:18:20 <yoctozepto> at least for some
15:18:23 <yoctozepto> ah ok
15:18:31 <yoctozepto> still, not in patch
15:18:36 <yoctozepto> osmanlicilegi not around hmm
15:19:15 <mgoddard> ok. Let's see how it progresses over the next few days
15:19:31 <openstackgerrit> Mark Goddard proposed openstack/kolla-ansible stable/train: Use valid_interfaces instead of os_interface for placement  https://review.opendev.org/696323
15:19:46 <mgoddard> https://launchpad.net/kolla-ansible/+milestone/9.0.0
15:20:01 <mgoddard> the open bug list does seem to be going down
15:20:48 <yoctozepto> osmanlicilegi: ping me by tomorrow - otherwise I assume you are not able to make it so I can pick it up
15:20:58 <yoctozepto> thanks
15:21:17 <mgoddard> are we planning to get enforce_scope to work for train?
15:21:27 <mgoddard> seems like a big change
15:21:41 <yoctozepto> I think he was around during the day
15:22:09 <yoctozepto> mgoddard: yeah, I'm afraid it could break in funny ways
15:22:16 <yoctozepto> I would like to but scared
15:22:18 <mgoddard> I'm tempted to remove the 9.0.0 milestone for that one
15:22:32 <mgoddard> it's a big enough patch that it looks like a feature
15:22:47 <yoctozepto> it's kind of to support a feature in train
15:23:02 <yoctozepto> "fix kolla-ansible to enable that feature" thing
15:23:08 <yoctozepto> borderline
15:23:47 <mgoddard> should we try to get https://bugs.launchpad.net/bugs/1844349 in ?
15:23:47 <openstack> Launchpad bug 1844349 in kolla-ansible ussuri "Neighbour table overflow problem with Neutron" [Medium,In progress] - Assigned to Pierre Riteau (priteau)
15:24:29 <priteau> That would be great. I am OK with removing the conntrack bits if that helps
15:24:32 <mgoddard> probably not urgent but osmanlicilegi said it helped
15:25:01 <priteau> I think I only took them from the charm code, it wasn't actually required on the deployment from where this fix originated
15:25:05 <yoctozepto> mgoddard, priteau: I am fine with either way, I just missed that there was any response on this, sorry ;-/
15:25:07 <mgoddard> the patch is W-1 currently priteau
15:25:25 <yoctozepto> taking unnecessary bits off is better as always
15:25:28 * priteau checking why it is like this
15:25:34 <mgoddard> W-1 Need to investigate if this is really relevant only for neutron-l3-agent.
15:25:37 <yoctozepto> if we all agree that less is more
15:25:40 <yoctozepto> (man less)
15:25:55 <priteau> Right, I wanted to check if it was needed only for neutron-l3-agent hosts
15:26:13 <yoctozepto> at least most relevant
15:26:22 <priteau> The problem is that the only environment where I can easily reproduce this is a production cloud :D
15:26:23 <yoctozepto> maybe get mnasiadka to ask neutron friends
15:26:32 <yoctozepto> priteau: :D
15:27:26 <hrw> priteau: perfect testing environment
15:27:40 <yoctozepto> closest to real world scenario
15:27:45 <hrw> priteau: at least when it fails lot of people tell you about it
15:27:59 <hrw> not that you will miss it etc
15:28:09 <yoctozepto> only pluses
15:28:47 <mgoddard> ok, if priteau can tidy up, we can try to get it into train
15:29:04 <mgoddard> I'll do another check of backports
15:29:05 <priteau> I am on it
15:29:08 <mgoddard> thanks
15:30:04 <mgoddard> I think rmq/erlang is the only thing holding the release at this point. Let's try to get an answer for that and get our GA out
15:30:28 <mgoddard> given the CentOS 8 migration we'll probably need some big changes to train down the line anyway
15:30:46 <hrw> once I sort out building erlang packages I will update patch with repos
15:30:53 <mgoddard> thanks gr
15:30:57 <mgoddard> hrw:
15:31:09 <mgoddard> let's move on to kayobe train release
15:31:39 <mgoddard> jovial has offered to do the release handle cranking
15:32:02 <mgoddard> we're hoping to be not too far behind kolla here
15:32:27 <mgoddard> we have a bunch of features listed on the whiteboard
15:32:49 <mgoddard> we don't really have time to do most of them though, so it's probably more of a list for Ussuri at this point
15:33:12 <openstackgerrit> Pierre Riteau proposed openstack/kolla-ansible master: [neutron] Adjust neighbour table thresholds  https://review.opendev.org/682664
15:33:41 <mgoddard> we have a train release etherpad: https://etherpad.openstack.org/p/kayobe-train-release
15:34:06 <mgoddard> #action mgoddard to update kayobe train release etherpad
15:34:35 <mgoddard> anything to add priteau?
15:35:25 <priteau> There are many patches that we can still merge, but you're right, the priorities will have to switch to Ussuri
15:35:36 <yoctozepto> so kayobe is pretty much mgoddard, priteau, jovial and dougsz?
15:36:13 <mgoddard> mostly. we have contributions from others at times including scottsol & co
15:36:39 <yoctozepto> I see
15:36:41 <mgoddard> those are the main active cores
15:36:56 <mgoddard> room for more if anyone is interested :)
15:37:23 <mgoddard> #topic Ussuri release planning
15:37:41 <mgoddard> Feels like we've made a good start on some features now
15:37:53 <mgoddard> centos8/py3/drop py2 in particular
15:38:04 <hrw> mgoddard: thanks for sorting out that in k-a
15:38:11 <mgoddard> no problem
15:38:43 <mgoddard> any changes in the centos8 container patch?
15:38:52 <hrw> I plan to submit something like https://review.opendev.org/695796 for k-a.
15:38:52 <patchbot> patch 695796 - kolla - test-requirements: bump pyflakes to get f-strings ... (MERGED) - 1 patch set
15:39:03 <hrw> mgoddard: no extra content yet
15:39:12 <hrw> mgoddard: centos team busy with 8.1 builds now
15:39:43 <mgoddard> hrw: what is failing in centos/source currently?
15:39:56 <mgoddard> that should be smoother than binary, in theory
15:40:27 <hrw> mgoddard: no ceph, openvswitch etc.
15:40:50 <mgoddard> hmm, OVS is an issue
15:41:08 <hrw> mgoddard: in general: check CI result. it shows nicely what fails
15:41:29 <mgoddard> yes. easier to ask you though, and we all find out :)
15:41:58 <hrw> mgoddard: last update was to just rebase it and let CI check how it goes
15:42:04 <mgoddard> I'm just thinking, could we get a single node ansible job to pass?
15:42:22 <hrw> aio?
15:42:24 <mgoddard> possibly with some fudging
15:42:26 <mgoddard> yes
15:42:35 <hrw> no idea to be honest
15:43:30 <mgoddard> https://review.opendev.org/#/c/692368/
15:43:30 <patchbot> patch 692368 - kolla - WIP: move to CentOS 8 in containers - 21 patch sets
15:43:36 <yoctozepto> someone mentioned testing in CI https://review.opendev.org/695192
15:43:36 <patchbot> patch 695192 - kolla-ansible - CI: Refactor a lot - 23 patch sets
15:43:45 <yoctozepto> testing, testing
15:43:48 <mgoddard> https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_30c/692368/21/check/kolla-build-centos-source/30c7c33/kolla/build/
15:44:03 <yoctozepto> I have the card "test swift"
15:44:09 <yoctozepto> because we have a bug pending on it
15:44:37 <hrw> yoctozepto: 'a lot' is very wrong commit message
15:44:53 <yoctozepto> hrwL very accurate though
15:45:14 <mgoddard> aio blockers: fluentd, glance-base, keystone-base, kolla-toolbox, neutron-base, nova-base, nova-libvirt, openvswitch-base, rabbitmq
15:45:44 <hrw> yoctozepto: can it get split into single small changes?
15:46:30 <hrw> yoctozepto: as now it looks like a way to sneak 7 separate changes in one commit
15:46:37 <yoctozepto> hrw: it is
15:46:43 <yoctozepto> it was meant to be testing qinling
15:46:52 <yoctozepto> but it was a pain so it got refactors in
15:46:58 <yoctozepto> now they are split
15:47:28 <hrw> yoctozepto: I mean to split that one 'split from qinling' patch
15:47:36 <yoctozepto> I know
15:48:08 <yoctozepto> extra work for not much gain, what can I say
15:48:43 <mgoddard> I would have preferred it to be split, but I have reviewed it now
15:49:03 <mgoddard> at least qinling is factored out now
15:50:20 <mgoddard> #topic Review priorities
15:50:42 <mgoddard> Same as usual really, train targeted bugs first, ussuri priorities second
15:50:59 <mgoddard> #topic Ceph orchestrator
15:51:04 <mgoddard> mnasiadka: you there?
15:51:17 <mnasiadka> yeah, just got to the keyboard
15:51:20 <mnasiadka> sorry for being late
15:51:28 <yoctozepto> (mgoddard, not seeing your review)
15:51:45 <mgoddard> (I reviewed before, not since your update)
15:51:47 <yoctozepto> mnasiadka: you had to rest from ml2 conf :-)
15:51:55 <yoctozepto> (I see)
15:51:58 <mgoddard> mnasiadka: Ceph orchestrator. Go!
15:52:06 <mnasiadka> Golang? ugh
15:52:20 <mnasiadka> So - cutting the story short - ceph-ansible's future is a bit not bright and shiny
15:52:28 <mnasiadka> What Ceph is going towards - is Ceph Orchestrator
15:52:50 <mnasiadka> so an abstraction layer between the operator/deployer and some deployment backends
15:53:06 <mnasiadka> in Octopus it will be the default deployment tool replacing ceph-deploy (and in future ceph-ansible)
15:53:32 <mgoddard> do you know which deployment backend will be recommendeD?
15:53:36 <hrw> march 2020, right?
15:53:39 <mnasiadka> So from my perspective - investing time in doing ceph-ansible integration with Kolla-Ansible is useless
15:53:55 <mnasiadka> mgoddard: the 'ssh' backend will be recommended - so it streams python scripts through ssh
15:54:09 <mnasiadka> hrw: march 2020 is the plan to release Octopus, yes
15:54:18 <mnasiadka> so it more or less fits into Ussuri timelines
15:54:21 <mgoddard> is the ssh backend different from ceph-deploy?
15:54:43 <mnasiadka> mgoddard: yeah, it doesn't require you to do all the pre-req jobs
15:54:51 <mnasiadka> the workflow is more or less:
15:54:52 <mgoddard> ok
15:55:00 <mnasiadka> 1) deploy 1 node ceph cluster (mon + mgr)
15:55:07 <mnasiadka> 2) enable ceph orchestrator module in mgr
15:55:14 <mnasiadka> 3) generate ssh keys and distribute them to nodes
15:55:24 <mnasiadka> 4) deploy more mons/mgrs for fully HA cluster
15:55:32 <mnasiadka> 5) scan for disks, deploy osds on them
15:55:43 <mnasiadka> (and add mds or rgw if you need them)
15:55:50 <mgoddard> 6) profit
15:56:05 <mnasiadka> yeah
15:56:17 <mnasiadka> so now the question is - what do we want to achieve from kolla-ansible perspective going forward?
15:56:32 <mnasiadka> I would say it would make sense to polish the external ceph
15:56:48 <mnasiadka> to remove current quirks and weirdness
15:57:29 <mgoddard> +1, and remain agnostic
15:57:36 <mnasiadka> I don't know if it makes any sense to integrate k-a with ceph orchestrator - it's should be up to the user to deploy it as he wants
15:57:42 <yoctozepto> +1 mgoddard
15:57:56 <mgoddard> we don't want to get hit by changing winds again
15:58:23 <mnasiadka> so from my perspective - maybe it's time to start moving towards removing kolla-ceph - or extracting it to separate project if there are maintainers interested in continuing
15:58:36 <mgoddard> we can always add support down the line if ceph orchestrator looks like it's here to stay, but it's hard to remove it
15:59:08 <mgoddard> mnasiadka: +1 for starting that conversation at least
15:59:09 <mnasiadka> mgoddard: I would probably prefer to write good docs instead of going that path... but there's still CI
15:59:23 <mnasiadka> I think we want to maintain a ceph-related CI, for all external ceph features
15:59:28 <mgoddard> yes
15:59:38 <mgoddard> I was going to ask what state your ceph-ansible patch is in
15:59:42 <mgoddard> could it be used for CI?
15:59:55 <mnasiadka> yeah, it could
16:00:04 <mnasiadka> the other option is to write a script leveraging the ceph orchestrator
16:00:25 <mnasiadka> or rework the ceph-ansible one - to be more frank
16:00:29 <mgoddard> we'd need to modify it to move the playbook to tests/
16:00:29 <yoctozepto> mgoddard, mnasiadka: regarding splitting - it seems to correlate with the general extensibility idea
16:00:48 <mgoddard> it would be a good test case for extensibility
16:00:54 <yoctozepto> indeed
16:01:07 <mgoddard> although I see the ansible side as being a separate CLI + site.yml rather than hooking in
16:01:11 <mnasiadka> mgoddard: that's not a problem - but I think I would prefer to remove the playbook (which is just an import from ceph-ansible) and add ceph orchestrator based deployment to the bash script
16:01:26 <mgoddard> mnasiadka: sounds like moar work
16:01:42 <mgoddard> it's taken quite a while to get ceph-ansible working, why not use it?
16:02:14 <mgoddard> and ceph orchestrator is still in development by the sound of it
16:02:16 <openstackgerrit> Merged openstack/kolla-ansible stable/train: Fix kolla_ansible python module check  https://review.opendev.org/696281
16:02:17 <openstackgerrit> Merged openstack/kolla-ansible master: Fix 'nova_cephx_raw_key' is undefined  https://review.opendev.org/695933
16:02:30 <mnasiadka> mgoddard: well, let's assume we can use it for now, I still have some $job plan to investigate ceph orchestrator - so we can move at feasible time to use it instead of ceph-ansible
16:03:02 <mgoddard> doesn't ceph orchestrator require octopus
16:03:04 <mgoddard> ?
16:03:05 <mnasiadka> mgoddard: so 1) adapt current ceph-ansible change to use as CI only 2) look in future to change it to use ceph-orchestrator
16:03:20 <mgoddard> ok, makes sense
16:03:26 <mnasiadka> mgoddard: it does, but RDO will build Octopus either way in Ussuri :)
16:03:39 <mgoddard> eventually yes
16:03:48 <mgoddard> anyway, we're out of time
16:04:07 <mgoddard> mnasiadka: can you kick off the conversation with openstack-discuss about ceph removal?
16:04:16 <mnasiadka> yes, will send a mail tomorrow
16:04:30 <mnasiadka> asking for contributors for extracting kolla-ceph
16:04:31 <mgoddard> maybe try to find some past committers & known users e.g. wangwei
16:04:43 <mnasiadka> should we give ourselves some deadline for answers?
16:04:47 <mnasiadka> Like end of December?
16:04:54 <mgoddard> yeah
16:05:01 <mnasiadka> ok, sounds good
16:05:08 <mgoddard> let's try to be a bit active in seeking people out though. This is a big move
16:05:28 <mnasiadka> It is, but we have sort of announced it in the beginning of Train cycle
16:05:50 <mgoddard> well we announced a supported migration path to ceph-ansible. This is not that
16:06:02 <mnasiadka> yeah
16:06:17 <mnasiadka> I'll rework the BP to deliver a CI-only implementation of ceph-ansible integration
16:06:25 <mgoddard> ok
16:06:39 <mnasiadka> we could raise a BP around ceph orchestrator and taking over a kolla-ceph deployment
16:07:00 <mnasiadka> but I doubt I will have time to work on it in a reasonable velocity, given my $job priorities
16:07:25 <mnasiadka> I can ask in the same mail to ML if there are volunteers :)
16:07:40 <mgoddard> that would be good.
16:07:47 <mgoddard> let's call it a day
16:07:51 <mgoddard> thanks all
16:07:54 <mgoddard> #endmeeting