15:00:10 <mgoddard> #startmeeting kolla
15:00:11 <openstack> Meeting started Wed Dec  4 15:00:10 2019 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:12 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:15 <mgoddard> #topic rollcall
15:00:15 <openstack> The meeting name has been set to 'kolla'
15:00:18 <mgoddard> \o
15:00:24 <yoctozepto> o/
15:00:26 <osmanlicilegi> o/
15:00:30 <scottsol> \o
15:00:30 <hrw> o/
15:03:30 <mgoddard> #topic agenda
15:03:41 <mgoddard> * Roll-call
15:03:43 <mgoddard> * Announcements
15:03:45 <mgoddard> * Review action items from last meeting
15:03:47 <mgoddard> * CI status
15:03:49 <mgoddard> * Train release planning
15:03:51 <mgoddard> * Ussuri release planning
15:03:54 <mgoddard> * Review priorities
15:03:56 <mgoddard> ** CI scenarios aka ETOOMANYJOBS
15:03:58 <mgoddard> #topic announcements
15:04:01 <mgoddard> Anyone have any?
15:04:41 <mgoddard> #topic Review action items from last meeting
15:04:44 <mgoddard> mgoddard to update kayobe train release etherpad
15:04:48 <mgoddard> I did not
15:04:51 <mgoddard> #action mgoddard to update kayobe train release etherpad
15:04:59 <mgoddard> #topic CI status
15:05:15 <mgoddard> Is kolla blocked by the tripleo job?
15:05:24 <osmanlicilegi> unfortunately
15:05:30 <hrw> other than rmq?
15:05:47 <osmanlicilegi> no, rmq only
15:05:58 <mgoddard> hmm, maybe not
15:06:00 <mgoddard> ok
15:06:02 <mgoddard> good :)
15:06:09 <mgoddard> Any issues?
15:06:40 <osmanlicilegi> nope
15:06:49 <mgoddard> ubuntu binary looks a bit broken
15:06:51 <mgoddard> http://zuul.openstack.org/builds?project=openstack%2Fkolla&branch=master&job_name=kolla-build-ubuntu-binary
15:07:32 <mgoddard> Unable to locate package senlin-health-manager
15:07:35 <mgoddard> busted
15:07:49 <yoctozepto> we had this recently merged, yeah
15:08:21 <mgoddard> should we disable these images for ubuntu binary for now then?
15:08:26 <mgoddard> anyone want to pick that up?
15:08:37 <osmanlicilegi> o/
15:08:48 <mgoddard> thanks
15:09:10 <mgoddard> #action osmanlicilegi to mark senlin images as unbuildable for ubuntu binary
15:09:28 <mgoddard> #undo
15:09:29 <openstack> Removing item from minutes: #action osmanlicilegi to mark senlin images as unbuildable for ubuntu binary
15:09:32 <mgoddard> #action osmanlicilegi to mark senlin images as unbuildable for ubuntu & debian binary
15:09:41 <mgoddard> debian busted too
15:09:51 <yoctozepto> mgoddard, osmanlicilegi ideally ask Erik about it
15:09:59 <yoctozepto> https://review.opendev.org/#/c/692948/3/docker/senlin/senlin-conductor/Dockerfile.j2
15:10:00 <patchbot> patch 692948 - kolla - Added senlin-conductor and senlin-health-manager (MERGED) - 3 patch sets
15:10:06 <yoctozepto> I think he wanted good :D
15:10:32 <mgoddard> yeah, we can add him to reviews. We need green CI though
15:10:34 <osmanlicilegi> got it
15:11:22 <mgoddard> #topic Train release planning
15:11:31 <mgoddard> RMQ/erlang
15:11:36 <mgoddard> So here's a funny thing
15:11:39 <yoctozepto> pinging him on #senlin
15:11:57 <mgoddard> I looked at the erlang version page, and the version we have already is fine for 3.7.10
15:12:11 <mgoddard> added this comment to the bug report:
15:12:13 <mgoddard> I think we are reading this wrong. RabbitMQ 3.7.10 supports 19.3.x to 21.x according to https://www.rabbitmq.com/which-erlang.html. So the versions we use are compatible.
15:12:59 <hrw> mgoddard: the problem is that we can not go beyond 3.7.10
15:13:00 <mgoddard> I think this removes the argument for pushing an erlang upgrade in train
15:13:14 <yoctozepto> mgoddard: I think we don't even have 19.3 on centos
15:13:22 <yoctozepto> the real reason behind this change
15:13:24 <mgoddard> hrw: that's right, but that's not a problem for train and earlier
15:13:35 <mgoddard> erlang-19.3.6.4-1.el7.x86_64
15:13:36 <yoctozepto> (no point staying too low though witj just 19.3)
15:13:42 <yoctozepto> hmm, odd
15:13:57 <yoctozepto> let me check again
15:14:32 <hrw> INFO:kolla.common.utils.rabbitmq: erlang                  aarch64 19.3.6.4-1.el7  delorean-master-testing   32 k
15:14:35 <mgoddard> IMO, we should leave this alone for train and upgrade erlang at the same time as rmq 3.8
15:14:49 <yoctozepto> mgoddard: they seem to have changed the support table
15:14:49 <hrw> +1
15:15:01 <mgoddard> (3.8 in ussuri)
15:15:08 <yoctozepto> they probably had a typo there
15:15:15 <yoctozepto> that's why it was installable
15:15:28 <mgoddard> yoctozepto: what did it say before?
15:15:33 <yoctozepto> 20.3
15:15:38 <mgoddard> hmm
15:15:41 <yoctozepto> like the row above
15:16:20 <osmanlicilegi> I remember it was 20 too
15:16:46 <yoctozepto> all the more reason to dump rmq and go qpid ;D
15:16:55 <yoctozepto> anyways
15:17:02 <yoctozepto> this makes this commit irrelevant for train
15:17:05 <yoctozepto> so no blockers
15:17:21 <yoctozepto> well, Joseph had some issue with rmq
15:17:32 <yoctozepto> but maybe something else was awry, who knows
15:17:42 <mgoddard> https://github.com/rabbitmq/rabbitmq-website/commit/d5c37acfe04876d5cdd62f3fc623fa35bb37601f#diff-4ae651de32ebda8d248ca50c4d6d0fb9
15:18:04 <mgoddard> looks like you're right
15:18:18 <yoctozepto> mgoddard: thank you
15:18:20 <osmanlicilegi> the patch is ready for 3.8 too, so lets go for ussuri
15:18:37 <openstackgerrit> Viktor Michalek proposed openstack/kolla-ansible master: Support for custom api-paste files  https://review.opendev.org/695054
15:18:44 <hrw> so what's left for train?
15:18:47 <yoctozepto> yeah, let's leave it because we are already lagging the release a bit
15:18:51 <yoctozepto> hrw: k-a patches
15:18:54 <yoctozepto> almost all are in
15:19:19 <yoctozepto> https://review.opendev.org/#/c/694476/
15:19:19 <patchbot> patch 694476 - kolla-ansible - Default to etcd3gw driver for etcd-based coordination - 4 patch sets
15:19:23 <mgoddard> https://launchpad.net/kolla-ansible/+milestone/9.0.0
15:19:24 <yoctozepto> https://review.opendev.org/#/c/697088/
15:19:24 <patchbot> patch 697088 - kolla - Install etcd3gw to fix Ubuntu binary tooz coordina... - 1 patch set
15:19:29 <yoctozepto> ah so it's kolla too
15:19:31 <yoctozepto> I forgot
15:20:05 <yoctozepto> masakari not going to happen
15:20:08 <yoctozepto> so unpin
15:20:34 <mgoddard> I need to look at that etcd business
15:21:04 <mgoddard> we're going back and forth in this area a bit
15:21:05 <yoctozepto> mgoddard: finally made it better, not cryptic
15:21:22 <mgoddard> do we know what other deployment projects use?
15:21:36 <yoctozepto> mgoddard: how so? I digested your point and did some tests
15:21:46 <yoctozepto> either etcd historically for v2
15:21:51 <yoctozepto> or etcd3gw for 3
15:22:00 <openstackgerrit> Chandan Kumar (raukadah) proposed openstack/kolla master: Added python3 packages for mellanox agent container  https://review.opendev.org/697236
15:22:15 <mgoddard> and we have etcdgw in centos images already?
15:22:19 <yoctozepto> otherwise redis or consul
15:22:25 <yoctozepto> mgoddard: yeah
15:22:30 <mgoddard> ok
15:23:09 <yoctozepto> python2-etcd3gw-0.2.4-6.el7.noarch
15:23:13 <yoctozepto> probably as a dep
15:23:27 <mgoddard> I wonder how compatible these drivers are
15:23:36 <yoctozepto> python2-cinder and openstack-cinder depend on it
15:23:48 <yoctozepto> no idea, etcd3 looks lame
15:23:49 <mgoddard> i.e. if you had etcd3 then we switch you to etcdgw, will it work?
15:24:02 <yoctozepto> mgoddard: well, it does work finally in CI
15:24:05 <mgoddard> python-etcd3 maintainer used to sit next to me in my office
15:24:14 <yoctozepto> mgoddard: what can I say
15:24:20 <yoctozepto> maybe tooz part is b0rken
15:24:22 <mgoddard> small world :)
15:24:26 <yoctozepto> indeed
15:25:12 <yoctozepto> our-project-wise they are interchangeable
15:25:19 <yoctozepto> both are limited to one endpoint
15:25:23 <mgoddard> I'm not seeing any hits for etcd3gw for deployment projects in http://codesearch.openstack.org/?q=etcd3gw&i=nope&files=&repos=
15:25:29 <yoctozepto> so we have to be lame and point to [0]
15:25:48 <yoctozepto> mgoddard: *coughs* devstack *coughs*
15:25:58 <yoctozepto> also, try etcd3+http
15:26:04 <yoctozepto> though codesearch fails with it
15:26:20 <yoctozepto> I did http://
15:26:27 <yoctozepto> and then browser for etcd3+http
15:26:37 <mgoddard> ok
15:26:49 <yoctozepto> http://codesearch.openstack.org/?q=etcd3%3A%2F%2F&i=nope&files=&repos=
15:26:57 <yoctozepto> now only tooz, us and vitrage (lol)
15:27:47 <yoctozepto> let's end the etcd drama :-)
15:27:54 <mgoddard> ok. I'll look at the reviews
15:28:07 <mgoddard> if we document, people can choose their own adventure
15:28:32 <yoctozepto> mgoddard: we have it kind-of documented in code and docs for designate
15:28:42 <yoctozepto> the bug for coordination is open for all improvements :-)
15:28:43 <mgoddard> ok
15:29:00 <mgoddard> Shall we aim for an RC2 by the weekend?
15:29:07 <hrw> +1
15:29:09 <yoctozepto> +!
15:29:14 <yoctozepto> =1
15:29:16 <mgoddard> I saw some 'bump versions' patches go by. Was there one for train?
15:29:17 <yoctozepto> mhm
15:29:35 <mgoddard> is that a yes?
15:29:52 <hrw> yes
15:29:58 <mgoddard> cool
15:29:58 <hrw> train/stein/rocky
15:30:02 <mgoddard> #topic Ussuri release planning
15:30:18 <hrw> we had user asking for monasca-ui bump so I did all for 3 branches
15:30:29 <openstackgerrit> Scott Solkhon proposed openstack/kolla-ansible master: Add also_notifies to Infoblox backend for Designate  https://review.opendev.org/697267
15:30:34 <yoctozepto> hrw: much appreciated
15:30:39 <mgoddard> +1
15:30:40 <yoctozepto> and for Ussuri we want a bot
15:30:45 <yoctozepto> who imitates hrw
15:31:01 <yoctozepto> or at least the version bumps...
15:31:05 <mgoddard> hrw: centos8 & RDO - you said they changed approach. What's going on?
15:31:11 <yoctozepto> :O
15:31:42 <hrw> for br in rocky stein train;do git checkout stable/$br; git up; ./tools/version-check.py;git commit -a -m"Bump versions for Openstack ${br}";git review;done
15:32:10 <hrw> mgoddard: no buildroot for centos8 so they started building all their deps in copr. to not waste time waiting and waiting
15:32:40 <yoctozepto> what a mess
15:33:28 <mgoddard> any ETA from them
15:33:30 <mgoddard> ?
15:33:42 <hrw> no
15:34:16 <mgoddard> ok
15:34:36 <hrw> heh. they had meeting an hour ago.
15:34:41 <hrw> moment, looking at log
15:34:58 <hrw> 14:09:29 <leanderthal> #topic CentOS8 Updates
15:34:59 <hrw> 14:09:30 <amoralej> i think ykarel added it
15:34:59 <hrw> 14:09:33 <amoralej> but anyway
15:34:59 <hrw> 14:09:46 <leanderthal> #info We are continuing building deps in copr
15:34:59 <hrw> 14:09:49 <ykarel> yes i did add
15:35:01 <hrw> 14:10:01 <leanderthal> #info     issues reported in https://review.rdoproject.org/etherpad/p/rebuild-deps-centos8
15:35:04 <hrw> 14:10:10 <rdogerrit> Soniya Vyas created rdo-infra/ci-config master: Changed the look of 'Compare jobs' button  https://review.rdoproject.org/r/23933
15:35:07 <hrw> 14:10:20 <leanderthal> #link https://review.rdoproject.org/etherpad/p/rebuild-deps-centos8
15:35:10 <hrw> 14:12:18 <amoralej> #info  If someone wants to try deps early can use http://38.145.34.66/test-el8/test-repos.repo
15:35:13 <hrw> 14:13:19 <amoralej> #info building DLRN builder for CentOS8 with master is in progress
15:35:16 <hrw> 14:14:03 <amoralej> #info we expect to start RDO Trunk CentOS8 bootstraping next week
15:35:36 <yoctozepto> nice
15:35:37 <hrw> http://eavesdrop.openstack.org/meetings/rdo_meeting___2019_12_04/2019/rdo_meeting___2019_12_04.2019-12-04-14.00.log.html
15:36:11 <mgoddard> thanks, good info
15:36:21 <mgoddard> hopefully getting closer then
15:36:52 <mgoddard> I wonder how much rdo-deps would help source builds
15:37:16 <mgoddard> https://copr-be.cloud.fedoraproject.org/results/%40openstack-sig/centos8-deps/centos-stream-x86_64/
15:37:26 <hrw> mgoddard: and how much for infra ones
15:37:49 <mgoddard> might be worth trying to point to that repo and seeing if it unlocks any more images
15:38:08 <openstackgerrit> Scott Solkhon proposed openstack/kolla-ansible master: Generate HAProxy configuration when upgrading from Rocky to Stein  https://review.opendev.org/697275
15:38:30 <mgoddard> we shouldn't have too many deps for source images, would be nice to get them building
15:39:21 <hrw> yep
15:39:28 <mgoddard> anyone want to try using that repo?
15:40:44 <hrw> (venv3) 16:40 (0s) hrw@puchatek:kolla$ time ~/devel/moje/docker-utils/kolla-build-images.sh centos "source binary" "" rdo-test
15:40:47 <hrw> started
15:40:56 <mgoddard> nice
15:41:19 <mgoddard> on the k-a side, I started a discussion on the ML about migration
15:41:23 <mgoddard> http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011130.html
15:41:43 <mgoddard> had a good response from mwhahaha about the tripleo approach
15:41:58 <mgoddard> I believe the plan is to have a Train version on CentOS8 after all the
15:42:00 <mgoddard> things get bootstrapped. Unfortunately the current target is trying to get
15:42:02 <mgoddard> master on centos8 with the time frame currently TBD.  I'm personally hoping
15:42:04 <mgoddard> really soon.
15:42:09 <mwhahaha> yea still waiting :(
15:42:56 <mgoddard> yeah hrw just checked rdo meeting logs, sounds like they're making some progress on master
15:43:25 <mwhahaha> yea last i heard was issues with a centos8 build root in cbs or something. So they wre going to bootstrap with copr or something
15:43:37 <mgoddard> yep
15:43:38 <hrw> mwhahaha: it is in progress
15:44:16 <mgoddard> so I think it makes sense for us to follow the tripleo approach here, which would require a slight modification of our proposal
15:44:49 <mgoddard> we would deploy centos 7 train containers on centos 7 hosts, and centos 8 train containers on centos 8 hosts
15:45:23 <mgoddard> so we'd need to support building containers for both releases in train
15:45:41 <mgoddard> and k-a would need to check the host's OS version and use the appropriate image ta
15:45:44 <mgoddard> *tag
15:45:54 <mgoddard> how does that sound?
15:45:57 <hrw> I just hope that it will not migrate to s/train/ussuri/ in plan
15:46:12 <yoctozepto> we need train on 8
15:46:13 <mgoddard> hrw: it can't really
15:46:15 <yoctozepto> period
15:46:32 <mgoddard> ussuri won't have centos 7 containers
15:46:56 <hrw> so far we do not have anythinf for c8 anyway
15:47:18 <mgoddard> it's a slightly awkward timing of kolla release, centos 8 release, and dropping py2
15:47:28 <hrw> sure, it is easier to migrate in one version of openstack
15:47:32 <yoctozepto> very awkward indeed
15:47:37 <mgoddard> anyways, let's press on
15:47:41 <yoctozepto> but py2 kina correlates
15:47:45 <yoctozepto> kinda*
15:47:46 <mgoddard> #topic Review priorities
15:47:47 <hrw> in worse case it would be centos7/train -> ubuntu/train -> */ussuri -> */v
15:47:56 <yoctozepto> lol
15:48:00 <mgoddard> hrw: lol
15:48:07 <mgoddard> let's pretend you didn't say that
15:48:09 <mgoddard> :)
15:48:20 <hrw> mgoddard: it is worst case scenario
15:48:33 <mgoddard> I could think of worse
15:48:37 <mgoddard> anyway...
15:48:44 <mgoddard> I think we know priorities
15:48:50 <mgoddard> train, then ussuri priorities
15:49:00 <mgoddard> #topic CI scenarios aka ETOOMANYJOBS
15:49:03 <yoctozepto> train etcd3
15:49:07 <yoctozepto> and that's it
15:49:11 <yoctozepto> oh yeah
15:49:15 * yoctozepto adding moar
15:49:57 <mgoddard> I saw the new senlin job proposal and it was the threshold for mw :)
15:50:01 <mgoddard> *me
15:50:12 <yoctozepto> :D
15:50:15 <mgoddard> it's great that we are adding more CI jobs
15:50:35 <hrw> too bad that infra starts to hate us due to this?
15:50:36 <yoctozepto> we have qinling, senlin and swift from me
15:50:41 <yoctozepto> all conflicting due to obvious reasons
15:50:45 <mgoddard> but I think we need to start using more scenarios
15:50:47 <yoctozepto> hrw: indeed
15:51:03 <mgoddard> where each scenario tests multiple services
15:51:32 <mgoddard> otherwise we will have 50 per-service jobs with 3 distro variants
15:51:41 <mgoddard> and we will not be invited back to opendev
15:51:44 <yoctozepto> mgoddard: that's my worry
15:51:52 <mgoddard> so we need to group things
15:51:53 <yoctozepto> though it might still not reduce that number too much
15:52:09 <yoctozepto> we are proposing to test zun with cinder
15:52:12 <yoctozepto> so the same job
15:52:19 <yoctozepto> but where the other would fit?
15:52:28 <yoctozepto> everywhere and nowhere :D
15:52:35 <mgoddard> along the same lines as the NFV scenario
15:52:44 <mgoddard> test multiple services in one job
15:52:54 <yoctozepto> yeah, zun and nfv are straight
15:53:11 <mgoddard> try to group them in a sane way
15:53:12 <yoctozepto> but it's the case that you need mistral and stuff for tracker
15:53:16 <yoctozepto> tacker*
15:53:20 <yoctozepto> (what a bad name)
15:53:31 <mgoddard> sure
15:54:23 <yoctozepto> point acked, had the same feeling so I strongly sympathize
15:55:21 <mgoddard> what's the best way to test two unrelated services in one job?
15:55:43 <yoctozepto> mgoddard: random match, most just way :D
15:55:44 <mgoddard> just enable them all and test sequentially?
15:55:55 <yoctozepto> but not fail immediatelly
15:56:01 <yoctozepto> still could fail right on deploy
15:56:02 <mgoddard> or setup one, test, destroy, setup two, test
15:56:08 <yoctozepto> and getting a job that runs 3 hours
15:56:20 <yoctozepto> mgoddard: then have separate jobs lol ;p
15:56:47 <yoctozepto> my ideas
15:56:50 <mgoddard> I think we spend a lot of time in the setup though
15:56:51 <yoctozepto> test what is popular
15:56:54 <yoctozepto> test what we use
15:56:58 <yoctozepto> test what is new
15:57:04 <yoctozepto> try to combine in real scenarios
15:57:16 <generalfuzz> then enable tls :)
15:57:20 <yoctozepto> mgoddard: which setup?
15:57:29 <yoctozepto> generalfuzz: yeah, it's in my review queue
15:57:33 <mgoddard> bootstrap servers etc
15:57:50 <mgoddard> haven't really profiled a job, that would be interesting to see where the time goes
15:57:52 <yoctozepto> just afraid it eats my dreams (encrypts them) and provides me nigthmares (wrong key for decryption)
15:58:10 <yoctozepto> mgoddard: so maybe that's the next step? :D
15:58:22 <yoctozepto> if we spend most in deploy/upgrade/reconfigure+tests
15:58:25 <mgoddard> maybe
15:58:31 <yoctozepto> then nothing to do
15:58:38 <mgoddard> I'm mostly going on the time taken for different jobs
15:58:54 <mgoddard> a job that deploys one service still seems to take a long time
15:59:30 <yoctozepto> mgoddard: hmm, well, it takes some time to do common and friends
15:59:31 <mgoddard> anyways, we're out of time
15:59:37 <yoctozepto> to check
15:59:48 <yoctozepto> thanks, mgoddard
15:59:55 <mgoddard> thank you all
16:00:00 <mgoddard> #endmeeting