15:00:49 <mgoddard> #startmeeting kolla
15:00:50 <openstack> Meeting started Wed Mar 10 15:00:49 2021 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:51 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:54 <openstack> The meeting name has been set to 'kolla'
15:00:59 <mgoddard> #topic rollcall
15:01:06 <yoctozepto> \o
15:01:17 <mgoddard> \-------o
15:01:59 <yoctozepto> o_
15:02:41 <priteau> \o/
15:02:48 <hrw> -°-
15:03:11 <hrw> yoctozepto: only while it is fresh.
15:03:33 <mgoddard> #topic agenda
15:03:36 <mgoddard> * Roll-call
15:03:39 <mgoddard> * Announcements
15:03:41 <mgoddard> ** PTG 19th - 23rd April, registration open | https://april2021-ptg.eventbrite.com | https://www.openstack.org/ptg/
15:03:43 <mgoddard> ** OpenStack feature freeze this week
15:03:45 <mgoddard> * Review action items from the last meeting
15:03:47 <mgoddard> * CI status
15:03:49 <mgoddard> * Review requests
15:03:51 <mgoddard> * PTG team signup http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html
15:03:53 <mgoddard> * Glance OSSN-0088 http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html
15:03:55 <mgoddard> * PoC: image build & test pipeline (https://review.opendev.org/c/openstack/kolla/+/777796 and https://review.opendev.org/c/openstack/kolla-ansible/+/777946)
15:03:57 <mgoddard> * Wallaby release planning
15:04:10 <mgoddard> #topic announcements
15:04:17 <Fl1nt> o/
15:04:43 <mgoddard> #info Project Teams Gathering (PTG) registration open
15:04:50 <mgoddard> #link https://april2021-ptg.eventbrite.com
15:04:55 <mgoddard> #link https://www.openstack.org/ptg/
15:05:07 <mgoddard> It will run from 19th-23rd April
15:05:21 <mgoddard> We can discuss it more later
15:05:33 <mgoddard> #info OpenStack feature freeze this week
15:06:03 <mgoddard> Which can only mean we are not far from Kolla feature freeze, so let's all get reviewing
15:06:09 <yoctozepto> yes, this is super painful
15:06:11 <yoctozepto> and sad
15:06:34 <mgoddard> Any other announcements?
15:07:55 <mgoddard> #topic Review action items from the last meeting
15:08:10 <mgoddard> wuchunyang to propose toscaparser in tacker requirements to fix NFV job
15:08:13 <mgoddard> mgoddard to write up options for CI registry
15:08:26 <mgoddard> I think the tacker issue turned out to be more complicated
15:08:33 <wuchunyang> tacker guys will fix this .
15:08:46 <mgoddard> #link https://bugs.launchpad.net/bugs/1918339
15:08:47 <openstack> Launchpad bug 1918339 in kolla "kolla-ansible-centos8-source-scenario-nfv Zuul test fails due to import error in tacker " [Undecided,New]
15:09:07 <mgoddard> I completed my action, we can discuss later
15:09:12 <mgoddard> #topic CI status
15:09:50 <mgoddard> observed on ussuri, victoria and master at least
15:09:53 <mgoddard> centos8-ceph-upgrade jobs seem to be retried 3 times only to fail in some weird way
15:09:55 <mgoddard> similarly other multinode centos8 jobs (like cells) fail: 'Ensuring config directories exist'
15:09:57 <mgoddard> yoctozepto suspects centos8.3
15:10:01 <mgoddard> Do we still see that, or should we scrub from the whiteboard?
15:10:15 <yoctozepto> I've seen that recently yes
15:10:25 <yoctozepto> but it's not that often
15:10:26 <mgoddard> ok
15:10:42 <yoctozepto> we can scrub it and remember it has happened previously
15:11:04 <yoctozepto> could be some cloud unhappy with centos
15:11:30 <yoctozepto> we know what the DISK_FULL are
15:11:37 <yoctozepto> it was due to swift spaces
15:11:38 <yoctozepto> for logs
15:11:58 <yoctozepto> but it was not close to 6.3GB
15:12:02 <yoctozepto> more like 630MB
15:12:13 <yoctozepto> still dumb to save and transfer
15:12:25 <yoctozepto> the responsible was libvirt debug
15:12:54 <yoctozepto> going to review now
15:13:42 <yoctozepto> mgoddard: https://review.opendev.org/c/openstack/kolla-ansible/+/779251
15:13:47 <yoctozepto> if you agree, I will just edit this
15:14:56 <yoctozepto> (or you can, whatever)
15:14:59 <mgoddard> yoctozepto: agree
15:15:02 <yoctozepto> ok
15:15:04 <yoctozepto> editing
15:15:14 <mgoddard> cool
15:15:50 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: Introduce nova_libvirt_logging_debug  https://review.opendev.org/c/openstack/kolla-ansible/+/779251
15:16:18 <yoctozepto> mgoddard: approve ^
15:16:21 <openstackgerrit> Radosław Piliszek proposed openstack/kolla-ansible master: Reduce number of logs and disable ara HTML report  https://review.opendev.org/c/openstack/kolla-ansible/+/777647
15:16:56 <mgoddard> yessir
15:17:43 <yoctozepto> :D
15:18:05 <mgoddard> ok, I think we're good for CI
15:18:17 <yoctozepto> agreed
15:18:26 <mgoddard> I did some 'tidying' of the whiteboard, it had developed some preamble
15:18:34 <mgoddard> (the CI section)
15:19:05 <mgoddard> #topic Review requests
15:19:08 <openstackgerrit> Merged openstack/kayobe stable/ussuri: Test building seed deployment images in the seed job  https://review.opendev.org/c/openstack/kayobe/+/774055
15:19:19 <mgoddard> Hit me with your code review requests (1 per person)
15:19:21 <openstackgerrit> Merged openstack/kayobe master: Wait for overcloud manage and provide state transitions  https://review.opendev.org/c/openstack/kayobe/+/775135
15:19:33 <openstackgerrit> Merged openstack/kayobe master: CI: Use cached cirros image for seed vm job  https://review.opendev.org/c/openstack/kayobe/+/773163
15:19:42 <priteau> CI in Kayobe is better (pull rate limit issues seem gone?) but not perfect (seeing occasional HTTP 520 from Galaxy)
15:19:43 <yoctozepto> I am pushing things with masakari atm so nothing specific from me
15:19:52 <priteau> (sorry, missed the topic change)
15:20:36 * yoctozepto slaps mgoddard around a bit with a large trout
15:20:41 <mgoddard> priteau: +1, although I think pull limits are just luck, I have seen them
15:20:55 <mgoddard> yoctozepto: that is not a review request
15:21:08 <yoctozepto> mgoddard: y, looks valid :D
15:23:15 <mgoddard> Nobody wants a review this week?
15:24:15 <kevko> do you think we will process mariadb,proxysql ... this week ?
15:24:28 <hrw> stream passed zuul so I assume that it will see revuiews
15:24:43 <mgoddard> kevko: probably not the whole patch chain
15:24:56 <mgoddard> kevko: but one can hope
15:24:58 <kevko> so, wallaby will be withou :/ ?
15:25:08 <yoctozepto> we are not freezing yet kevko
15:25:18 <kevko> but it is behind the door
15:25:45 <openstackgerrit> Pierre Riteau proposed openstack/kayobe stable/ussuri: Update IPA docs and test build with extra-hardware  https://review.opendev.org/c/openstack/kayobe/+/779811
15:25:59 <yoctozepto> worry not
15:26:12 <mgoddard> I would suggest that we aim to merge 2 of the proxysql patches per week
15:26:26 <mgoddard> hopefully that should get us there by feature freeze
15:26:26 <yoctozepto> starting from the bottom
15:26:34 <mgoddard> yes
15:26:47 <mgoddard> mariadb sharding patch looks close
15:27:15 <mgoddard> https://review.opendev.org/c/openstack/kolla-ansible/+/766952
15:27:18 <kevko> in few hours i will get into and check again
15:27:55 <mgoddard> the next one is https://review.opendev.org/c/openstack/kolla-ansible/+/770618/18
15:28:12 <kevko> btw, i have some small patches i want to review ..let me check it
15:28:56 <kevko> https://review.opendev.org/c/openstack/kolla-ansible/+/775627 <<
15:29:23 <headphoneJames> Working on the let's encrypt functional test, getting closer
15:30:34 <mgoddard> sounds good headphoneJames
15:30:37 <mgoddard> Let's move on
15:30:47 <mgoddard> #topic PTG team signup
15:30:54 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020915.html
15:31:13 <mgoddard> We need to book some time for the PTG
15:31:57 <mgoddard> Our 'usual' approach is 4 hours on Monday, 4 hours on Tuesday, and 2 hours on Wednesday for Kayobe
15:32:33 <mgoddard> We usually take the 13:00+ UTC slot
15:32:44 <yoctozepto> I won't be able to dedicate that much time this time
15:32:54 <yoctozepto> perhaps we could do 2 hours per deliverable
15:33:52 <mgoddard> I was thinking, should we try a slot in the 4:00 - 8:00 UTC window?
15:34:18 <hrw> any fits me
15:34:35 <mgoddard> Not because I like waking up early, but it might attract some people in Asian timezones
15:34:39 <mgoddard> wuchunyang: around?
15:35:56 <priteau> Keep in mind that clocks will have changed by then
15:36:21 <mgoddard> good point
15:36:27 <priteau> So 4-8 UTC will be 5-9 UK time and 6-10 CEST
15:37:16 <yoctozepto> yes
15:37:18 <mgoddard> so how do we decide? Should I send an email to openstack-discuss to gauge interest?
15:37:26 <yoctozepto> ++
15:37:49 <mgoddard> #action mgoddard send an email to openstack-discuss to gauge interest in the 'early' PTG slot
15:38:51 <mgoddard> #topic Glance OSSN-0088
15:38:55 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020947.html
15:39:07 <mgoddard> who dis?
15:40:24 <mgoddard> Is there anything for us to do here?
15:40:40 <priteau> Ship secure policies by default?
15:41:23 <mgoddard> I would probably prefer glance to do that
15:41:23 <wuchunyang> hi
15:41:29 <yoctozepto> it was me
15:41:34 <yoctozepto> yes, I agree
15:41:43 <yoctozepto> but it was thrown at the deployment projects
15:41:45 <mgoddard> hi wuchunyang
15:41:53 <yoctozepto> how should we approach this issue?
15:41:55 <yoctozepto> hi wuchunyang
15:42:21 <wuchunyang> which issue ? i was out just now
15:43:29 <hrw> wuchunyang: PTG in Asian time
15:43:30 <mgoddard> wuchunyang: I think I pinged you earlier about the PTG. I'll send an email to openstack-discuss, you can reply there
15:43:46 <yoctozepto> wuchunyang: the issue question was not targeted at you ;-)
15:44:09 <wuchunyang> ok.. i will read the mail..
15:44:17 <mgoddard> So far we have avoided modifying default policies
15:44:36 <mgoddard> The email is targeted at operators rather than deployment tools
15:45:24 <priteau> Maybe this is a topic to explore more generally
15:45:33 <mgoddard> Should we make an exception for this specific issue?
15:45:39 <priteau> Hardened Kolla-Ansible
15:45:59 <yoctozepto> well, it looks crappy to let it have this hole by default
15:45:59 <Fl1nt> I suggest we don't change the default policy as if it really is a security issue upstream services will do it, and if not it means that an optional part of the service and should be let up to the operators
15:46:07 <priteau> Could be a global variable that would change some settings to be more secure, with the downside of breaking some features
15:46:34 <yoctozepto> all right, we have got the first topic for ptg
15:46:37 <priteau> yoctozepto: It's a hole that some deployments may rely on
15:46:43 <Fl1nt> we shouldn't, it will have impact on upgrades/updates as some side effects will arise.
15:46:46 <yoctozepto> priteau: yes, unfortunately
15:47:16 <yoctozepto> yeah, stable/upgrades is something I would prefer not to touch indeed
15:47:23 <mgoddard> I think if it were as simple as just changing the policy, glance would do it
15:47:45 <mgoddard> they are just as capable as us of making the change
15:47:50 <yoctozepto> yes, it would have to be behind a flag like priteau said
15:48:25 <Fl1nt> but are they calling metadef APIs ?
15:48:35 <Fl1nt> ^what are
15:48:44 <mgoddard> I suppose I'm not against having it behind a flag
15:49:14 <yoctozepto> Fl1nt: https://docs.openstack.org/api-ref/image/v2/metadefs-index.html
15:49:15 <Fl1nt> yeah something like INVALID_GLANCE_OSSN0088
15:49:17 <mgoddard> What I didn't mention earlier in the PTG section is that I created an etherpad
15:49:18 * hrw out
15:49:19 <mgoddard> #link  https://etherpad.opendev.org/p/kolla-xena-ptg
15:49:36 <Fl1nt> aaaaah metadefinitions, got it
15:49:36 <mgoddard> does someone want to add this topic?
15:49:39 <priteau> Fl1nt: https://docs.openstack.org/glance/latest/#metadata-definitions
15:50:09 <yoctozepto> Fl1nt: yeas
15:50:29 <yoctozepto> someone added it
15:50:36 <yoctozepto> (not me)
15:50:38 <mgoddard> me
15:52:04 <Fl1nt> what I don't understand, is that from the mailing list responses, it seems to only be a default policy shape issue on glance metadef api, so glance should just add the usual is_admin filter, but did they responded to the mail?
15:52:52 <mgoddard> I don't see any responses
15:53:18 <mgoddard> We're running out of time
15:53:23 <mgoddard> Let's discuss at the PTG
15:53:30 <Fl1nt> yep
15:53:32 <mgoddard> #topic PoC: image build & test pipeline
15:53:42 <mgoddard> #link https://etherpad.opendev.org/p/docker-pull-limits
15:53:51 <mgoddard> I wrote up some options in there
15:53:56 <mgoddard> See 3a. and 3b.
15:55:29 <mgoddard> I think we don't have time to discuss this topic properly today
15:55:45 <mgoddard> But please read the etherpad, and we can discuss next week
15:56:10 <mgoddard> #topic Open discussion
15:57:11 <priteau> I would like to know what is our plan for Docker iptables management in Xena
15:57:27 <priteau> It was announced to be disabled in Victoria I think?
15:57:32 <Fl1nt> woot?
15:57:46 <priteau> But it wasn't
15:57:47 <priteau> https://review.opendev.org/c/openstack/kolla-ansible/+/751795
15:57:58 <Fl1nt> ah you're talking about the infamous ebtable thingy ?
15:58:16 <priteau> and https://review.opendev.org/c/openstack/kolla-ansible/+/689870
15:58:32 <priteau> No, I am talking about Docker manipulating iptables rules on its own
15:58:35 <yoctozepto> yeah, we are late
15:58:40 <priteau> Particularly, changing the default FORWARD chain policy
15:58:53 <priteau> Which can cause issues in some environments
15:59:02 <Fl1nt> Don't use use host networking model ?
15:59:26 <Fl1nt> s/use/you/
15:59:51 <priteau> Of course, but it still has an impact
16:00:07 * yoctozepto went away
16:00:16 <mgoddard> Good question priteau
16:00:38 <mgoddard> "As Michal pointed out, this breaks kolla-build unless you set  network_mode = host in kolla-build.conf."
16:01:08 <Fl1nt> yes, I do on my own.
16:01:54 <mgoddard> I think I had problems when I tested it here: https://review.opendev.org/c/openstack/kolla-ansible/+/751982
16:02:33 <mgoddard> I can try it again
16:03:04 <mgoddard> we do now have network_mode = host in kolla-build.conf
16:03:13 <mgoddard> anyway, time's up
16:03:16 <mgoddard> Thanks all
16:03:21 <mgoddard> #endmeeting