15:01:37 <mgoddard> #startmeeting kolla
15:01:38 <openstack> Meeting started Wed Jun 17 15:01:37 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:42 <openstack> The meeting name has been set to 'kolla'
15:01:43 <openstack> yoctozepto: Error: Can't start another meeting, one is in progress.  Use #endmeeting first.
15:01:50 <yoctozepto> cool, openstack ;-)
15:01:52 <mgoddard> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi
15:02:04 <yoctozepto> o/
15:02:10 <yoctozepto> ah, no
15:02:14 <mgoddard> patience yoctozepto
15:02:15 <yoctozepto> false start
15:02:23 <yoctozepto> sorry mgoddard :-(
15:02:24 <mgoddard> We're live
15:02:29 <mgoddard> #topic rollcall
15:02:30 <yoctozepto> oh noez
15:02:31 <priteau> o/
15:02:33 <mgoddard> \o
15:02:35 <yoctozepto> is it being broadcast? :D
15:02:36 <yoctozepto> o/
15:02:37 <hrw> o/
15:02:41 <yoctozepto> dougsz
15:02:45 <yoctozepto> jovial[m]
15:02:47 <headphoneJames> o/
15:02:49 <dougsz> o/
15:02:50 <yoctozepto> headphoneJames
15:02:56 <yoctozepto> headphoneJames: oh, sorry
15:02:57 <hrw> yoctozepto: #chair mgoddard
15:03:09 <yoctozepto> hrw: yeah, I noticed
15:03:10 <hrw> yoctozepto: for next time
15:03:24 <yoctozepto> hrw: seconds matter ;-(
15:03:34 <yoctozepto> hrw: I'd chair mgoddard anyhow
15:04:12 <hrw> ;D
15:04:42 <mgoddard> #topic agenda
15:04:50 <mgoddard> * Roll-call
15:04:52 <mgoddard> * Announcements
15:04:54 <mgoddard> ** Kolla klub meeting tomorrow 15:00 UTC
15:04:56 <mgoddard> * Review action items from last meeting
15:04:58 <mgoddard> * CI status
15:05:00 <mgoddard> * Ussuri release planning (kayobe)
15:05:02 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:05:04 <mgoddard> * Kolla klbu
15:05:06 <mgoddard> #topic announcements
15:05:13 <mgoddard> #info Kolla klub meeting tomorrow 15:00 UTC
15:05:37 <mgoddard> #info Ussuri release blockers added to whiteboard L105
15:06:01 <mgoddard> #info PTG recordings and actions updated on etherpad
15:06:04 <mgoddard> #link https://etherpad.opendev.org/p/kolla-victoria-ptg
15:06:11 <mgoddard> Any others?
15:06:14 <yoctozepto> (kolla klbu sounds like cthulhu ;D )
15:06:37 <hrw> We are close to huge speedup for aarch64 ci
15:06:43 <yoctozepto> hrw: yeah, I like it <3
15:06:47 <hrw> #link https://marcin.juszkiewicz.com.pl/2020/06/15/opendev-ci-speed-up-for-aarch64/
15:07:31 <yoctozepto> #info we can test kvm in CI now
15:07:37 <yoctozepto> #link https://review.opendev.org/735454
15:07:37 <patchbot> patch 735454 - kolla-ansible - CI: Add Ubuntu KVM job - 1 patch set
15:08:45 <hrw> yoctozepto: as 'baremetal' like?
15:09:05 <yoctozepto> hrw: it's via nested virt
15:09:19 <yoctozepto> hrw: but essentially we can test our kvm code paths
15:09:31 <mgoddard> #topic Review action items from last meeting
15:09:32 <hrw> ok, can discuss it later
15:09:47 <mgoddard> mgoddard to collect actions from PTG
15:09:48 <mgoddard> jovial[m] to revert the switch to ussuri patch on master, bump previous release to ussuri
15:09:50 <mgoddard> jovial[m] to polish up https://review.opendev.org/#/c/709689/
15:09:50 <patchbot> patch 709689 - kayobe - Add framework to deploy user-defined containers on... (MERGED) - 18 patch sets
15:09:58 <mgoddard> mgoddard did his
15:10:06 <mgoddard> jovial[m] did his
15:10:18 <mgoddard> #topic CI status
15:10:53 <hrw> centos8 is in problem
15:11:20 <hrw> openvswitch has issue with deps. Newer version without dpdk dependency is on a way
15:11:24 <yoctozepto> how so?
15:11:26 <yoctozepto> oh my
15:11:39 <hrw> master is not affected as we use dlrn
15:11:39 <mgoddard> hrw: so it's out of our control?
15:11:49 <hrw> but ussuri suffer
15:12:01 <hrw> mgoddard: yes
15:12:32 <yoctozepto> and stein?
15:12:42 <yoctozepto> only virtualenv thingy?
15:12:53 <yoctozepto> gimme da patchaz
15:13:26 <mgoddard> virtualenv affects kolla too, mnasiadka proposed a patch
15:14:14 <yoctozepto> ah, we have been fixing k-a back then
15:15:00 <mgoddard> https://review.opendev.org/736162
15:15:00 <patchbot> patch 736162 - kolla (stable/train) - CI: Install python dependencies - 4 patch sets
15:15:20 <yoctozepto> train is fried independently
15:15:34 <mgoddard> how so?
15:17:03 <mgoddard> https://review.opendev.org/#/c/736038/ to unblock rocky k-a
15:17:04 <patchbot> patch 736038 - kolla-ansible (stable/rocky) - CI: Install python dependencies - 1 patch set
15:17:17 <yoctozepto> mgoddard: hrw just told us about ovs failure on c8
15:17:30 <mgoddard> oh, right. I thought you meant something else
15:18:14 <yoctozepto> np
15:18:29 <mgoddard> I think k-a is actually green everywhere now then
15:18:35 <yoctozepto> hmm, mnasiadka's fix seems to fail on k-a 'Ensure tox is installed'
15:18:59 <yoctozepto> Cannot uninstall 'virtualenv'
15:19:04 <yoctozepto> doh, doh, doh, doh
15:19:16 <mgoddard> kayobe is broken by the virtualenv issue too. Working on https://review.opendev.org/#/c/735921/ to resolve
15:19:17 <patchbot> patch 735921 - kayobe (stable/train) - CI: don't run virtualenv on an existing virtualenv - 2 patch sets
15:20:03 <hrw> I digged in other projects so missed most of failures
15:20:23 <mgoddard> ok. I think we've covered them all
15:20:32 <yoctozepto> ay-ay, we broke k-a for kolla
15:20:39 <yoctozepto> k-a jobs in kolla conflict
15:20:46 <yoctozepto> due to tox install
15:21:01 <mnasiadka> yoctozepto: working on it ;)
15:21:09 <yoctozepto> mnasiadka: ok
15:21:21 <yoctozepto> then we are covered
15:21:30 <yoctozepto> k-a to fix for k
15:21:34 <yoctozepto> virtualenv in k
15:21:43 <yoctozepto> and ovs out of our control train/ussuri
15:22:01 <yoctozepto> thx mgoddard for the summary on wb ;d
15:22:24 <yoctozepto> k-a looks green apart from that, no?
15:22:27 <mgoddard> in summary: thanks infra!
15:22:33 <mgoddard> (and centos)
15:23:04 <yoctozepto> mgoddard: oh, we could prepare better with vanilla nodes
15:23:15 <mnasiadka> External deps are always painful
15:23:23 <yoctozepto> well, anyway, let's move on ;-)
15:23:28 <yoctozepto> now the env is much cleaner
15:23:32 <yoctozepto> so closer to users
15:23:38 <yoctozepto> I'm +2 with that
15:24:06 <mgoddard> #topic Ussuri release planning (kayobe)
15:24:34 <mgoddard> priteau, jovial[m], dougsz: assemble
15:24:43 <dougsz> ACK
15:24:58 <priteau> HERE
15:24:59 <jovial[m]> hey
15:24:59 <mgoddard> Fix with network named external: https://storyboard.openstack.org/#!/story/2007814
15:25:01 <mgoddard> https://review.opendev.org/#/c/735551
15:25:02 <patchbot> patch 735551 - kayobe - Fix kolla-ansible config when a network is named '... - 1 patch set
15:25:03 <mgoddard> Fix baremetal commands: https://storyboard.openstack.org/#!/story/2007797
15:25:05 <mgoddard> https://review.opendev.org/735197
15:25:06 <patchbot> patch 735197 - kayobe - Fix kayobe baremetal compute commands - 1 patch set
15:25:07 <mgoddard> Missing image regexes: https://storyboard.openstack.org/#!/story/2007796
15:25:09 <mgoddard> https://review.opendev.org/#/c/735181/
15:25:09 <patchbot> patch 735181 - kayobe - Update image regexes - 3 patch sets
15:25:11 <mgoddard> https://review.opendev.org/#/c/735329
15:25:12 <patchbot> patch 735329 - kayobe - Update feature flag defaults - 2 patch sets
15:25:13 <mgoddard> Fix Inspection: https://storyboard.openstack.org/#!/story/2007795
15:25:15 <mgoddard> https://review.opendev.org/735148
15:25:16 <patchbot> patch 735148 - kayobe - Fix ironic inspector store endpoint configuration - 3 patch sets
15:25:17 <mgoddard> Certificates? https://storyboard.openstack.org/#!/story/2007679
15:25:19 <mgoddard> https://review.opendev.org/#/c/728453/
15:25:20 <patchbot> patch 728453 - kayobe - Copy custom certificates - 4 patch sets
15:25:21 <mgoddard> IPA builder https://storyboard.openstack.org/#!/story/2007070
15:25:23 <mgoddard> https://review.opendev.org/735115
15:25:23 <patchbot> patch 735115 - kayobe - IPA: Switch to IPA builder and CentOS 8 - 5 patch sets
15:25:26 <mgoddard> this is our list of release blockers. If you think we've missed anything, please add
15:25:43 <dougsz> I think many should now be on their way..
15:26:12 <jovial[m]> is there a label I can use in gerrit to get that list?
15:26:27 <priteau> label:Review-Priority+1
15:26:37 <jovial[m]> thanks :)
15:26:47 <priteau> https://review.opendev.org/#/q/label:Review-Priority%252B1+status:open+project:openstack/kayobe
15:26:51 <Fl1nt> Hi everyone!
15:27:04 <mgoddard> hi Fl1nt
15:27:06 <hrw> Fl1nt: we are in a meeting.
15:27:12 <hrw> Fl1nt: for ~0.5h
15:27:16 <mgoddard> seems the only one in contention is https://review.opendev.org/#/c/728453/
15:27:16 <patchbot> patch 728453 - kayobe - Copy custom certificates - 4 patch sets
15:27:22 <Fl1nt> yep, saw the header
15:27:27 <yoctozepto> hi Fl1nt; the discussion is on kayobe now
15:27:34 <Fl1nt> thx
15:27:39 <priteau> mgoddard: I'll take another look at the certificate one
15:27:54 <mgoddard> priteau: thanks. Would be good to close it today if possible
15:28:04 <mgoddard> then all those can be backported and we're good
15:28:24 <priteau> Before the next CI breakage...
15:28:29 <mgoddard> indeed :)
15:28:35 <mgoddard> I think ussuri is ok at least
15:28:48 <mgoddard> train & earlier not so much
15:29:13 <mgoddard> Anything else on the kayobe release?
15:29:37 <priteau> Not on the Ussuri release specifically
15:29:42 <mgoddard> ok
15:29:48 <priteau> Stable CI needs work indeed
15:29:55 <dougsz> +1, nothing here
15:30:21 <jovial[m]> nothing from me
15:30:22 <mgoddard> If anyone sees requests for help from bsanjeewa, please respond. They're trying to get started with kayobe
15:30:35 <jovial[m]> is he on iRC too?
15:30:55 <priteau> I've replied to one or two storyboard queries
15:30:57 <jovial[m]> I've only seen the kayobe issues on storyboard
15:30:58 <mgoddard> yes, but in APAC timeone
15:31:17 <jovial[m]> I'll keep an eye out
15:31:20 <mgoddard> cool
15:31:27 <mgoddard> Anything else in Kayobe town?
15:32:04 <mgoddard> priteau: did you have something else?
15:32:12 <priteau> Nope
15:32:20 <mgoddard> ok
15:32:26 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:32:54 <mgoddard> octavia create loadbalancer failed due service_auth: https://bugs.launchpad.net/bugs/1882643
15:32:54 <openstack> Launchpad bug 1882643 in kolla-ansible victoria "octavia create loadbalancer failed due service_auth" [High,In progress] - Assigned to Mark Goddard (mgoddard)
15:32:54 <mgoddard> https://review.opendev.org/727160
15:32:55 <patchbot> patch 727160 - kolla-ansible - Switch octavia to use service project in service_auth - 2 patch sets
15:32:57 <mgoddard> Prometheus: https://bugs.launchpad.net/bugs/1882460
15:32:57 <openstack> Launchpad bug 1882460 in kolla-ansible victoria "Prometheus task "Copying over prometheus config file" has been broken since 10.0.0.0rc1" [High,In progress] - Assigned to Radosław Piliszek (yoctozepto)
15:32:58 <mgoddard> fix is proposed and being reviewed - yoctozepto
15:33:00 <mgoddard> https://review.opendev.org/735529
15:33:01 <patchbot> patch 735529 - kolla-ansible - Fix issues with Prometheus config generation - 1 patch set
15:33:02 <mgoddard> Cinder OSM bug: https://review.opendev.org/735617
15:33:03 <patchbot> patch 735617 - kolla-ansible - Remove max count from Cinder online schema migration - 3 patch sets
15:33:04 <mgoddard> kolla_admin_openrc_cacert: https://review.opendev.org/733743
15:33:05 <patchbot> patch 733743 - kolla-ansible - Replace internal and external VIP CA with root CA (MERGED) - 5 patch sets
15:33:06 <mgoddard> and dependency https://review.opendev.org/#/c/735874/
15:33:06 <patchbot> patch 735874 - kolla-ansible (stable/ussuri) - Generate Root CA for Self-Signed Certificates (MERGED) - 1 patch set
15:33:08 <mgoddard> Keystone fernet follow up: https://review.opendev.org/707080
15:33:08 <patchbot> patch 707080 - kolla-ansible - Fix fernet bootstrap and key distribution - follow up - 14 patch sets
15:33:10 <mgoddard> Revert nova mod_wsgi: https://review.opendev.org/#/c/735821
15:33:10 <patchbot> patch 735821 - kolla (stable/ussuri) - Revert "Enable mod_wsgi in Nova API container" - 1 patch set
15:33:24 <mgoddard> No release blockers listed for kolla. The above are kolla-ansible
15:33:36 <yoctozepto> yes, kolla looks fine to release
15:33:56 <yoctozepto> except for the images being broken by ovs
15:34:01 <mgoddard> Please focus your reviews on those
15:34:07 <mgoddard> minor
15:35:37 <mgoddard> actually the nova mod_wsgi one is kolla
15:35:51 <mgoddard> Does anyone have other bugs or patches which should block the release?
15:36:47 <yoctozepto> mgoddard: it would be nice to get kvm friendliness fix too
15:36:54 <yoctozepto> mgoddard: to get rid of high-heat bug
15:37:01 <yoctozepto> mgoddard: but not really a blocker
15:37:10 <headphoneJames> depends if you want to include fixes for https://review.opendev.org/#/c/736088/
15:37:10 <patchbot> patch 736088 - kolla-ansible - DNM: enable TLS for all jobs pt 2 - 1 patch set
15:38:18 <mgoddard> I was mostly trying to include regressions
15:38:33 <mgoddard> bugs that exist in previous releases we can follow up with
15:38:45 <headphoneJames> copy that
15:39:06 <mgoddard> (I was more replying to yoctozepto, but works for headphoneJames too)
15:39:51 <mgoddard> octavia is a nasty one, so I included that
15:40:07 <mgoddard> I agree it would be good to get the KVM fix in if we can though
15:41:00 <mgoddard> Anything else for kolla/kolla-ansible?
15:41:03 <yoctozepto> it's quite simple... and tested!
15:41:10 <yoctozepto> looks fine to me
15:41:20 <mgoddard> #topic Kolla klub
15:41:32 <mgoddard> So tomorrow we have the meeting
15:41:33 <yoctozepto> kblu
15:41:45 <yoctozepto> klbu*
15:41:47 <mgoddard> https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_4_Sw/edit#
15:41:57 <yoctozepto> I made a typo in a typo ;/
15:42:04 <mgoddard> Agenda:
15:42:06 <mgoddard> Virtual PTG
15:42:08 <mgoddard> https://etherpad.opendev.org/p/kolla-victoria-ptg
15:42:10 <mgoddard> Open discussion
15:42:12 <mgoddard> Please propose some topics
15:42:20 <mgoddard> Does anyone want to volunteer to help summarise the PTG?
15:42:45 <mgoddard> we could split general/kolla/kolla-ansible/kayobe
15:42:46 <hrw> it is tomorrow at same time as today?
15:42:50 <mgoddard> ~ 5 minutes each
15:42:53 <mgoddard> hrw: ys
15:42:55 <hrw> adding to calendar to not forget again
15:42:59 * yoctozepto overloaded, prefers to focus on CI and reviews too
15:43:06 <mgoddard> ok
15:43:17 <mgoddard> hrw: maybe you could cover the kolla discussions?
15:44:18 <hrw> mgoddard: I feel like last 2-3 weeks were kollaless here
15:44:37 <mgoddard> priteau: if you are available could you handle kayobe?
15:44:43 <mgoddard> hrw: what do you mean?
15:45:14 <priteau> mgoddard: sure
15:45:17 <mgoddard> thanks
15:45:53 <hrw> mgoddard: I had several other projects occupying my time more than kolla so I was looking at kolla at random times. have to find time to reread ptg notes to remember myself what we agreed on
15:47:34 <mgoddard> hrw: do you not want to do it?
15:47:56 <hrw> yep. no idea will manage to join meeting due to some activities here
15:48:13 <hrw> sorry
15:48:13 <mgoddard> ok
15:48:27 <mgoddard> #topic Open discussion
15:48:49 <Fl1nt> Ok, got one on this
15:49:17 <Fl1nt> How is CentOS 8 supported so far?
15:49:28 <hrw> Fl1nt: you mean?
15:49:40 <yoctozepto> Fl1nt: very well
15:49:44 <yoctozepto> Fl1nt: stupendously!
15:49:53 <yoctozepto> Fl1nt: marvelously!
15:49:57 <Fl1nt> Did anyone tried to build a CentOS8 based deploiement recently?
15:50:12 <mgoddard> yes, we're running a few
15:50:15 <yoctozepto> Fl1nt: does weekend work?
15:50:20 <mgoddard> still early days
15:50:27 <yoctozepto> Fl1nt: currently ovs is broken
15:50:36 <yoctozepto> Fl1nt: is that what you are referring to?
15:50:42 <Fl1nt> 'cause I think we may have an issue with Openvswitch dependencies
15:50:47 <yoctozepto> this is out of our scope
15:50:50 <yoctozepto> ah, so this is
15:51:05 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: [docs] Update quickstart for Ussuri+  https://review.opendev.org/735894
15:51:06 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: CI: use venv only (and not virtualenv)  https://review.opendev.org/736276
15:51:16 <hrw> Fl1nt: you use ussuri?
15:51:27 <Fl1nt> was trying to build ^^
15:51:36 <yoctozepto> hrw: it's broken in either train and ussuri, no?
15:51:36 <hrw> Fl1nt: known issue ;(
15:51:46 <Fl1nt> both of them yeah
15:51:49 <hrw> yoctozepto: running c8/train build now to check.
15:51:56 <yoctozepto> hrw: ok, th
15:51:57 <yoctozepto> thx
15:52:10 <hrw> Fl1nt: centos team works on solving it
15:52:20 <openstackgerrit> Mark Goddard proposed openstack/kayobe stable/rocky: CI: don't run virtualenv on an existing virtualenv  https://review.opendev.org/736278
15:52:52 <openstackgerrit> Mark Goddard proposed openstack/kayobe stable/stein: CI: don't run virtualenv on an existing virtualenv  https://review.opendev.org/736280
15:52:58 <hrw> c8/t c8/u is broken
15:53:05 <hrw> c8/master works due to dlrn
15:53:20 <openstackgerrit> Mark Goddard proposed openstack/kayobe stable/train: CI: don't run virtualenv on an existing virtualenv  https://review.opendev.org/735921
15:53:39 <Fl1nt> ok nice, is there any ETA (link?) on this ovs topic?
15:53:48 <hrw> https://cbs.centos.org/koji/buildinfo?buildID=30269 shows that it is built
15:54:16 <hrw> and https://review.rdoproject.org/r/#/c/28107/ is enabler
15:54:36 <hrw> https://review.opendev.org/#/c/735912/ too
15:54:36 <patchbot> patch 735912 - puppet-nova (stable/ussuri) - Prepare for 16.4.0 release (MERGED) - 1 patch set
15:54:38 <Fl1nt> checked 2h ago, didn't work on CentOS-8.1-1911
15:55:21 <Fl1nt> Maybe not yet backported on train so?
15:55:23 <hrw> we wait for puppet-nova 16.4.0 tarball release
15:55:59 <hrw> 16.3.0 fails to work on centos 8.2
15:56:14 <hrw> so it is yak shaving as usual
15:56:35 <yoctozepto> oh, so we depend on puppet, great
15:56:36 <yoctozepto> :D
15:56:45 <Fl1nt> all right, I'll switch on CentOS7 in the meantime as I need our new region deployed by the end of the week.
15:56:47 <hrw> yoctozepto: we depend on opendev tarball generator
15:56:57 <yoctozepto> hrw: just kidding ;d
15:57:13 <yoctozepto> puppet triggers me ;p
15:57:29 <hrw> Fl1nt: you may alter images and give them url to newer openvswitch packages. or make own repo with them
15:58:51 <Fl1nt> you mean, use the ussuri package on train repo?
15:59:03 <mgoddard> I need to prepare for another meeting. I'll leave this open for a few minutes
15:59:16 <hrw> mgoddard: feel free to #end
15:59:22 <hrw> we may continue anyway
15:59:28 <Fl1nt> yep, no worries
15:59:41 <hrw> Fl1nt: openvswitch one? sure
16:00:18 * yoctozepto leaving now, see ya
16:00:45 <Fl1nt> see you @yoctozepto
16:01:59 <Fl1nt> @hrw, I'll wait for the backport, I can't mess with the local repo, it's used by multiple teams ^^
16:02:05 <hrw> ok
16:04:12 <Fl1nt> we agree that this latest build is about that kind of issue right? http://paste.openstack.org/show/PCN6AuSrV9rVYOdGwszl/
16:04:35 <Fl1nt> caused by the dpdk framework missing.
16:05:08 <hrw> yes
16:05:32 <Fl1nt> all right, perfect
16:06:43 <mgoddard> #endmeeting