15:02:19 <yoctozepto> #startmeeting kolla
15:02:20 <openstack> Meeting started Wed Sep 16 15:02:19 2020 UTC and is due to finish in 60 minutes.  The chair is yoctozepto. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02:24 <openstack> The meeting name has been set to 'kolla'
15:02:26 <yoctozepto> #chair mgoddard
15:02:27 <openstack> Current chairs: mgoddard yoctozepto
15:02:28 <mloza> the keystone container just keep crashing with following error http://paste.openstack.org/show/wAUq2YLHPyefomvU8AAr/
15:02:33 <yoctozepto> #topic Roll-call
15:02:35 <yoctozepto> o/
15:02:38 <priteau> o/
15:02:40 <osmanlicilegi> o/
15:03:00 <mgoddard> \o
15:03:37 <Fl1nt> \o
15:03:56 <mgoddard> #topic agenda
15:03:58 <mgoddard> * Roll-call
15:04:00 <mgoddard> * Announcements
15:04:02 <mgoddard> ** Kolla feature freeze fast approaching: Sep 28 - Oct 02
15:04:04 <mgoddard> ** Submit Virtual PTG topic proposals: https://etherpad.opendev.org/p/kolla-wallaby-ptg
15:04:06 <mgoddard> ** TC & PTL elections coming up: http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017306.html
15:04:08 <mgoddard> * Review action items from the last meeting
15:04:10 <mgoddard> * CI status
15:04:12 <mgoddard> * Victoria release planning
15:04:14 <mgoddard> * Bump ansible?
15:04:24 <mgoddard> #topic announcements
15:04:34 <mgoddard> #info Kolla feature freeze fast approaching: Sep 28 - Oct 02
15:05:00 <mgoddard> Now would be a good time to review outstanding priority patches
15:05:30 <mgoddard> #info Submit Virtual PTG topic proposals
15:05:36 <mgoddard> #link https://etherpad.opendev.org/p/kolla-wallaby-ptg
15:05:42 <mgoddard> I created ^
15:05:47 <yoctozepto> I fixed
15:06:28 <Fl1nt> we can submit topics on this page right?
15:06:47 <mgoddard> Yes. Please add discussion topic proposals under 'Discussion Topic Proposals'
15:07:03 <mgoddard> And if you would like to attend, please add your name to the list
15:07:18 <mgoddard> and register https://october2020ptg.eventbrite.com
15:07:47 <mgoddard> #info TC & PTL elections coming up
15:07:59 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017306.html
15:09:08 <mgoddard> Now is a good time to think about if you are interested in running for the technical committee (TC) or a Project Team Lead (PTL)
15:09:33 <mgoddard> Nominations start next week
15:10:26 <mgoddard> Anyone is able to nominate themselves for PTL, including Kolla PTL
15:10:42 <mgoddard> Any other announcements?
15:11:31 <mgoddard> #topic Review action items from the last meeting
15:11:34 <mgoddard> There were none
15:11:37 <Fl1nt> For now, I'm still learning the whole process but sure
15:11:48 <mgoddard> #topic CI status
15:12:24 <mgoddard> Looks like we have some issues in kolla with influx repos
15:12:32 <mgoddard> Has anyone looked? yoctozepto?
15:12:45 <yoctozepto> mgoddard: fails from ovh
15:12:54 <yoctozepto> must be east coast having broken cdn endpoint
15:14:08 <mgoddard> ok
15:14:38 <mgoddard> hopefully it will sort itself out
15:15:04 <mgoddard> if still broken tomorrow let's pursue
15:15:51 <yoctozepto> ok
15:16:00 <mgoddard> we have neutron pinned still. Any ideas if they've fixed it?
15:16:18 <yoctozepto> it's not pin, we don't run some migrations
15:16:20 <mgoddard> not pinned
15:16:22 <mgoddard> right
15:16:34 <yoctozepto> I'll pick it up later this week
15:16:38 <mgoddard> is there a bug?
15:16:48 <yoctozepto> yup, there is a report
15:17:15 <yoctozepto> https://bugs.launchpad.net/bgpvpn/+bug/1894056
15:17:16 <openstack> Launchpad bug 1894056 in networking-bgpvpn "DB migration broken" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)
15:17:38 <yoctozepto> it seems neutron hanging on it
15:19:00 <mgoddard> ok
15:19:13 <mgoddard> we also have no bifrost in stein and rocky
15:19:30 <Fl1nt> Isn't rocky freezed and EOL ?
15:19:33 <mgoddard> can someone confirm I have this right
15:19:36 <mnasiadka> well, we have no new bifrost, there are some old builds that work
15:19:50 <yoctozepto> mnasiadka: ++
15:19:51 <mgoddard> we need to get a fix into bifrost for LANG
15:20:04 <mgoddard> but bifrost CI is broken due to http://lists.openstack.org/pipermail/openstack-discuss/2020-September/016999.html
15:20:05 <yoctozepto> but bifrost CI seems fried per Dmitry
15:20:10 <yoctozepto> indeed
15:20:18 <mgoddard> ok, updating whiteboard
15:20:45 <openstackgerrit> Merged openstack/kayobe master: Update shebang for Python 3 script  https://review.opendev.org/752241
15:21:01 <mgoddard> hmm, actually its merged to stein
15:21:08 <mgoddard> https://review.opendev.org/748599
15:21:08 <patchbot> patch 748599 - kolla (stable/stein) - Fix ironic build issues - LANG related (MERGED) - 3 patch sets
15:21:30 <jovial[m]> for what it is worth, I built a stein centos7 based bifrost container by applying https://review.opendev.org/#/c/749014/
15:21:30 <mgoddard> and rocky
15:21:30 <patchbot> patch 749014 - bifrost (stable/stein) - [Train and older] Fix several CI problems - 2 patch sets
15:21:44 <mnasiadka> mgoddard: at least ironic works thanks to that, but not bifrost
15:21:51 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/ussuri: Update shebang for Python 3 script  https://review.opendev.org/752261
15:22:04 <jovial[m]> also had to re-enable the bifrost base image in build.py
15:22:24 <mgoddard> mnasiadka: ah, I see
15:22:35 <jovial[m]> I'm not sure if there is a warning printed, but to me it was very unobvious what was going on
15:22:48 <mgoddard> this is the bifrost one: https://review.opendev.org/#/c/748646/
15:22:49 <patchbot> patch 748646 - bifrost - Change LANG in install-deps.sh to en_US.UTF-8 (MERGED) - 1 patch set
15:23:56 <mgoddard> I suppose a nasty workaround would be to modify the script from our dockerfile
15:24:24 <mgoddard> would be much better to fix bifrost CI though
15:24:30 <mnasiadka> jovial[m]: I'm pretty sure kolla-build does print out UNBUILDABLE at some point
15:25:18 <jovial[m]> not near the bottom - that is for sure!
15:25:37 <jovial[m]> it may have been buried in there somewhere
15:25:52 <yoctozepto> jovial[m] probably talking about that ugly json that prints out
15:25:59 <yoctozepto> with everything summarised
15:26:04 <yoctozepto> "summarised"
15:26:08 <yoctozepto> ;-)
15:26:41 <mgoddard> anyone want to try fixing bifrost stable CI?
15:27:27 <mgoddard> guess not
15:27:31 <mgoddard> ok, onto kolla-ansible
15:27:55 <mgoddard> ubuntu binary still broken IIRC, let's see if it fixes itself as it often does
15:28:35 <Fl1nt> Ubuntu binary can't retrieve the package from CDN
15:28:42 <Fl1nt> CDN/REPO mirror
15:28:43 <hrw> o\
15:29:14 <mgoddard> Looks like the bifrost issue is now resolved
15:29:36 <mgoddard> kayobe
15:30:07 <mgoddard> I think the inspector API issue is resolved now priteau & jovial[m]?
15:30:22 <priteau> It is
15:30:36 <mgoddard> good
15:30:47 <priteau> At least while the ironic TLS patch is reverted
15:31:24 <priteau> I saw James submitted it again: https://review.opendev.org/#/c/751543/
15:31:25 <patchbot> patch 751543 - kolla-ansible - Add support for encrypting Ironic API - 3 patch sets
15:31:40 <priteau> Careful, Zuul says +1 but the ironic jobs are red
15:31:49 <mgoddard> yes, this time without wsgi for inspector
15:31:57 <mgoddard> not working yet
15:32:17 <headphoneJames> Still working on it
15:32:20 <mgoddard> cool
15:32:23 <mgoddard> #topic Bump ansible?
15:32:29 <mgoddard> Changing the running order a bit here
15:32:49 <mgoddard> It occurred to me we might want to bump supported ansible versions before release
15:32:59 <mgoddard> 2.8-2.9 -> 2.9-2.10
15:33:10 <Fl1nt> yeah in order to get the deprecation message at least ^^
15:33:16 <mgoddard> mnasiadka was tentative
15:33:55 <mnasiadka> well, I was just pessimistic, that it might cause some work
15:33:59 <mgoddard> 2.9 supports the throttle keyword, which could some in handy
15:34:19 <priteau> I would like to see 2.9 minimum, that would be nice
15:34:20 <mgoddard> it could cause work. hopefully we would find out sooner rather than later
15:34:20 <yoctozepto> I vote for it
15:34:22 <Fl1nt> @mnasiadka, why that? 2.8 and 2.9 are that much appart?
15:34:27 <priteau> 2.10 is a bit fresh thoguh
15:34:37 <yoctozepto> let's test 2.10 on all jobs once
15:34:37 <Fl1nt> I vote for it too ^^
15:34:38 <mnasiadka> Fl1nt: Ansible is extremely good at introducing new bugs
15:34:43 <yoctozepto> and see how it goes
15:34:51 <yoctozepto> mnasiadka: they perfected the art of it
15:35:04 <yoctozepto> and they be reluctant about fixing any existing
15:35:09 <mnasiadka> yeah
15:35:09 <yoctozepto> mgoddard's fix still laying there
15:35:12 <mnasiadka> so let's test
15:35:14 <priteau> mnasiadka: they do it on stable branches too, so we might as well get new features ;-)
15:35:17 <mnasiadka> and see what comes out of it
15:35:25 <yoctozepto> priteau: "features"
15:35:27 <Fl1nt> @mnasiadka, I suppose that I too used to for being surprised anymore ^^
15:35:30 <mnasiadka> priteau: "security features"
15:35:37 <mnasiadka> by obscurity
15:35:39 <mnasiadka> :)
15:35:53 <wuchunyang> ansible 2.10  not works now for kolla-ansible . i have encountered a probelm that gather facts failed.
15:36:15 <mgoddard> wuchunyang: good to know. Do you know what the problem is?
15:37:13 <mgoddard> does anyone want to pick up this task?
15:37:15 <wuchunyang> i remember  that use --limit  will take the mistake
15:37:21 <mgoddard> ok
15:37:23 <yoctozepto> mgoddard: bumping ansible?
15:37:26 <mgoddard> yes
15:37:28 <yoctozepto> my pleasure
15:37:31 <priteau> What is the rationale for supporting two minor Ansible versions? if there were issues with 2.10, could we still require >2.9,<2.10?
15:37:38 <yoctozepto> LET THE ENGINES FALL
15:37:51 <yoctozepto> priteau: that's what we would do I guees
15:37:54 <yoctozepto> guess*
15:38:10 <mgoddard> priteau: I suppose it gives a bit of flexibility in case one breaks
15:38:14 <yoctozepto> xD
15:38:23 <mgoddard> or doesn't work for a particular use case
15:38:53 <mgoddard> #action yoctozepto to try bumping ansible to 2.9-2.10
15:38:54 <yoctozepto> the fact that we come up with these ideas means there is something wrong about ansible in the first place
15:39:06 <mgoddard> software breaks
15:39:21 <mgoddard> you could say there is something wrong about software
15:39:32 <yoctozepto> this one likes it too much though
15:39:34 <yoctozepto> ;d
15:39:38 <mgoddard> fall back to 2.9-2.9
15:39:41 <yoctozepto> could be just popularity
15:39:43 <Fl1nt> ansible not being the only software to breaks and software being horrible to get stable, ansible remains the better alternative to nothing for now ^^
15:40:05 <yoctozepto> Fl1nt: you are an ubuntu folk, y u no use charms? :D
15:40:17 <yoctozepto> the juju magic :-)
15:40:35 <mgoddard> #topic Victoria release planning
15:40:42 <Fl1nt> I'm not an ubuntu folk at all ^^ rather the opposite and charms fails everytime ^^
15:40:45 <mgoddard> Final push now
15:41:01 <yoctozepto> :-(
15:41:04 <mgoddard> ~2 weeks until feature freeze
15:41:30 <mgoddard> let's go through priorities on the whiteboard
15:41:32 <mgoddard> #link https://etherpad.opendev.org/p/KollaWhiteBoard
15:41:41 <mgoddard> L203
15:41:46 <yoctozepto> well, if it's any consolation, masakari just got its first and last feature this cycle
15:41:50 <yoctozepto> so we ain't that bad
15:41:53 <mgoddard> lol
15:42:11 <mgoddard> hrw: infra images
15:42:25 <mgoddard> can we land it?
15:42:52 <mgoddard> I gave my +2 to the big one
15:43:33 <yoctozepto> :O
15:43:34 <mgoddard> we need to do a zuul CI dance to land the others
15:44:08 <mnasiadka> and I'm fixing the kolla_docker updates to support healthchecks
15:44:23 <mgoddard> ok
15:44:36 <mnasiadka> would appreciate a review on the big change to playbooks to add this feature
15:44:48 <mnasiadka> https://review.opendev.org/#/c/676389/
15:44:49 <patchbot> patch 676389 - kolla-ansible - Use Docker healthchecks for core services - 67 patch sets
15:44:51 <priteau> I've started looking into adding pull-retag-push to kolla-build
15:45:31 <mgoddard> wuchunyang: thanks for comments on octavia patch
15:46:11 <mgoddard> wuchunyang: do you think you will have time to get it merged?
15:46:34 <wuchunyang> yes, i have time
15:46:48 <mgoddard> great
15:46:57 <mgoddard> there is a CI test for magnum & octavia here: https://review.opendev.org/652030
15:46:58 <patchbot> patch 652030 - kolla-ansible - CI: add magnum scenario, also covering octavia - 8 patch sets
15:47:11 <mgoddard> would like to merge that first so that we have some test coverage of octavia
15:48:02 <mgoddard> ideally it would be updated to actually create and test a load balancer, but that may be a stretch for Victoria
15:48:43 <mnasiadka> well, once zuul says +1 ;)
15:48:53 <Fl1nt> @mnasiadka, really nice feature about Docker healthcheck, same thing with the infra image @hrw
15:48:57 <mgoddard> I did a big review of keystone federation https://review.opendev.org/695432
15:48:58 <patchbot> patch 695432 - kolla-ansible - Add support to OpenID Connect Authentication flow - 34 patch sets
15:49:17 <wuchunyang> maybe we can not create a lb for test ,because it needs amphora image.
15:49:22 <mgoddard> Fl1nt: I'm sure mnasiadka would appreciate your review on it :)
15:49:34 <Fl1nt> I'm doing it ^^
15:49:42 <mgoddard> wuchunyang: correct. octavia must test their own code though?
15:49:58 <Fl1nt> And I'd like to get a quick review on this minor pip patch so it can be merged and backported to train at some point ^^
15:50:04 <mgoddard> it would be nice if they published amphora images, but I don't think they do
15:50:17 <Fl1nt> they don't and it's better
15:50:40 <wuchunyang> yes , there is no amphora image for downloading .
15:50:47 <priteau> https://tarballs.opendev.org/openstack/octavia/test-images/
15:50:51 <hrw> Fl1nt: want to continue work on it?
15:50:53 <priteau> would this help?
15:50:54 <Fl1nt> amphora images should be considered as "customer images" as you have so many way to tweak it ^^
15:51:06 <Fl1nt> @hrw, on what? image?
15:51:06 <mgoddard> nice find Fl1nt
15:51:10 <mgoddard> sorry, priteau
15:51:21 <priteau> I haven't tried them, no idea if they work
15:51:32 <hrw> Fl1nt: infra stuff
15:51:33 <wuchunyang> priteau looks good .
15:51:39 <johnsom> We created those images for OSA testing. They are fine for gate tests, but should not be used in deployments
15:51:56 <Fl1nt> @hrw, depends on which timeline do we have
15:52:02 <hrw> ;D
15:52:13 <mgoddard> let's focus on getting the ansible running first, then we can do an integration test
15:52:16 <priteau> It's used in some other services: http://codesearch.openstack.org/?q=test-only-amphora&i=nope&files=&repos=
15:52:16 <mgoddard> johnsom: ack
15:52:45 <wuchunyang> mgoddard  it seems that we can create a lb for gate now .
15:52:51 <mgoddard> yes
15:53:21 <mgoddard> I have some runes to make a backend with a cirros image somewhere
15:53:33 <mgoddard> anyone familiar with keystone federation?
15:53:49 <mgoddard> if so, https://review.opendev.org/695432 awaits
15:53:49 <patchbot> patch 695432 - kolla-ansible - Add support to OpenID Connect Authentication flow - 34 patch sets
15:53:54 <Fl1nt> A bit yes, depends on your backend @mgoddard
15:54:32 <yoctozepto> mgoddard: unfortunately, federation/octavia are tough for me because I'd have to base reviews off the common/other knowledge only
15:55:01 <yoctozepto> hence why everything usually gets higher priority :-(
15:55:26 <mnasiadka> mgoddard: not yet familiar, but it might change in coming weeks ;-)
15:55:28 <mgoddard> Fl1nt: I'm sure they would appreciate your experience
15:55:40 <mgoddard> that patch has been around for a while, would be nice to land it
15:55:43 <Fl1nt> I'm looking into, but I'm more experienced on ADFS
15:55:54 <yoctozepto> any comments welcome, Fl1nt
15:56:16 <mgoddard> tempting to merge it as experimental, if we can confirm it does not break anything
15:56:32 <Fl1nt> @yoctozepto, oki doo, I'll do it.
15:56:43 <mgoddard> leaves room to change the interface if necessary e.g. for other protocols
15:56:59 <mnasiadka> yeah, there's an option to fix in a bugfix post-release - and if we don't merge it, then it's Wallaby content...
15:57:59 <yoctozepto> mnasiadka: just need to mark 'experimental' / 'subject to change' as mgoddard said
15:58:03 <mgoddard> health checks we covered
15:58:42 <mgoddard> scalability still has a few patches open
15:59:06 <mgoddard> https://review.opendev.org/739122 and https://review.opendev.org/739112 would be good to land
15:59:06 <patchbot> patch 739122 - kolla-ansible - Performance: use a single config file for fluentd - 3 patch sets
15:59:08 <patchbot> patch 739112 - kolla-ansible - Performance: use a single config file for logrotate - 5 patch sets
15:59:20 <mgoddard> as common tasks run on all nodes
15:59:25 <mgoddard> it's a big win
15:59:49 <mgoddard> and anything in https://review.opendev.org/#/q/topic:bp/performance-improvements+branch:master+status:open
16:00:09 <mgoddard> TLS we have neutron and ironic in progress
16:00:21 <yoctozepto> argh, one has merge conflict
16:00:26 <yoctozepto> and it was mgoddard+2
16:00:31 <mgoddard> https://review.opendev.org/#/c/728448/ did have a yoctozepto +2
16:00:31 <patchbot> patch 728448 - kolla-ansible - Support TLS encryption of RabbitMQ client-server t... - 9 patch sets
16:00:33 <yoctozepto> I'll try to dig it up
16:00:44 <mgoddard> maybe if he adds it back it will tempt someone to approve
16:01:10 <mgoddard> kayobe
16:01:24 <yoctozepto> mgoddard: might be
16:01:35 <mgoddard> I will look at https://review.opendev.org/#/c/734867/ today or tomorrow
16:01:35 <patchbot> patch 734867 - kayobe - Support multiple environments from a single config... - 1 patch set
16:02:06 <mgoddard> would be nice to get agreement on https://review.opendev.org/#/c/748582/
16:02:07 <patchbot> patch 748582 - kayobe - docs: Add skeleton for scenario docs - 1 patch set
16:02:23 <mgoddard> then we can start adding scenarios
16:02:37 <mgoddard> and I think that's all
16:03:23 <mgoddard> If there are other patches you will like to land in the release, please make it known, and we can assess
16:03:36 <mgoddard> #topic Open discussion
16:03:37 <Fl1nt> yes I do ^^
16:03:50 <mgoddard> Kolla kall tomorrow. Do we have an agenda?
16:04:02 <yoctozepto> mgoddard: continue bugs
16:04:11 <Fl1nt> @mgoddard, I would like to get this one included https://review.opendev.org/#/c/751787/
16:04:11 <patchbot> patch 751787 - kolla - Improve pip install process for offline deployment. - 1 patch set
16:04:14 <yoctozepto> I'm open to using the time more effectively though
16:04:35 <yoctozepto> we might also agree to spend it on doing reviews
16:04:38 <mgoddard> yoctozepto: ok. Do we need to do anything more to hand out tasks for those docs improvements?
16:04:39 <yoctozepto> (self-paced)
16:05:08 <yoctozepto> mgoddard: we should do something about them but due to time constraints I doubt they have high enough priority to look at
16:05:13 <mgoddard> Fl1nt: added to review queue
16:05:21 <Fl1nt> \o/ THX
16:05:33 <mgoddard> yoctozepto: there may be some easy wins
16:05:38 <wuchunyang> https://blueprints.launchpad.net/kolla-ansible/+spec/add-hosts-command
16:06:12 <wuchunyang> is there  anyone work on this bp?
16:06:32 <yoctozepto> mgoddard: well, we can discuss just that then
16:06:32 <mgoddard> wuchunyang: not that I am aware of
16:06:33 <Fl1nt> you can already do it @wuchunyang using deploy with limit
16:06:40 <yoctozepto> and spend rest of time on other reviews
16:06:43 <yoctozepto> or actually doing docs
16:07:03 <mgoddard> wuchunyang: we do lack documentation in this area though
16:07:12 <mgoddard> I think priteau has some notes on it
16:07:32 <wuchunyang> Fl1nt  at the bootstrap  we will change hosts  and need restart some docker
16:08:05 <wuchunyang> i want to add-host  without reboot docker.
16:08:31 <priteau> I don't think I have notes specifically on the /etc/hosts issue
16:08:39 <yoctozepto> I'd be good to have this option
16:08:49 <yoctozepto> I'm just using dns to solve these issues
16:09:06 <wuchunyang> dns is a good way, for now
16:09:29 <Fl1nt> DNS is the only real way actually ^^
16:09:38 <mgoddard> there is another way
16:09:41 <yoctozepto> well, it adds a pain point
16:09:52 <wuchunyang> i tested on vm ,, we can create kolla-toolbox with host network
16:09:53 <Fl1nt> yeah, but DNS is an infra requirement that anyone should have.
16:09:53 <mgoddard> we no longer need hostnames for computes
16:10:08 <mgoddard> libvirt migration by IP was fixed
16:10:13 <mgoddard> so rabbitmq is the last issue
16:10:23 <wuchunyang> and create other containers with net: container:kolla-toolbox
16:10:28 <Fl1nt> rmq is able to use gethostbyname() now
16:10:30 <mgoddard> so only hosts running rabbitmq need hostnames
16:10:31 <yoctozepto> Fl1nt: you'd be surprised but there are company-wide /etc/hosts
16:10:52 <Fl1nt> Don't tell me such things @yoctozepto my day was so bright so far!
16:11:05 <mgoddard> ok, we are over time
16:11:10 <mgoddard> Thanks everyone
16:11:14 <mgoddard> Back to your reviews!
16:11:17 <mgoddard> #endmeeting