15:01:41 <mgoddard> #startmeeting kolla
15:01:42 <openstack> Meeting started Wed Nov  4 15:01:41 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:46 <openstack> The meeting name has been set to 'kolla'
15:02:15 <mgoddard> #topic rollcall
15:02:20 <mgoddard> \o
15:02:25 <mgoddard> mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi bbezak
15:02:50 <hrw> ]o]
15:06:12 <yoctozepto> o/
15:06:33 <yoctozepto> taking a break after a day-long meeting
15:06:50 * yoctozepto might be sluggish to respond
15:06:58 <yoctozepto> ping me
15:07:46 <JamesBenson> \o
15:09:43 <mgoddard> #topic agenda
15:09:50 <mgoddard> * Roll-call
15:09:52 <mgoddard> * Announcements
15:09:54 <mgoddard> ** Vote for Kolla Wallaby priorities https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:09:56 <mgoddard> * Review action items from the last meeting
15:09:58 <mgoddard> * CI status
15:10:00 <mgoddard> * Victoria release planning
15:10:02 <mgoddard> * Wallaby PTG actions
15:10:04 <mgoddard> #topic announcements
15:10:06 <mgoddard> #info Vote for Kolla Wallaby priorities
15:10:15 <mgoddard> #link https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:10:53 <mgoddard> Not too many votes on there yet, please add your name on up to 12 items
15:11:00 <mgoddard> #topic Review action items from the last meeting
15:11:41 <mgoddard> undeprecate VMware
15:11:56 <mgoddard> did it happen?
15:12:20 <mgoddard> https://review.opendev.org/#/c/747512/
15:12:20 <patchbot> patch 747512 - kolla-ansible - Revert VMware deprecation (MERGED) - 2 patch sets
15:12:49 <mgoddard> I guess so
15:12:58 <mgoddard> #topic CI status
15:13:01 <mgoddard> mnasiadka:
15:13:07 <mnasiadka> Oh boy
15:13:18 <mgoddard> Seems like we have some issues
15:13:29 <mnasiadka> Kolla - I think curator strikes again, but need to verify
15:13:32 <mgoddard> is the whiteboard up to date?
15:13:49 <mnasiadka> Yes, up to date
15:13:59 <mgoddard> great
15:14:02 <mnasiadka> Especially with the Nova rmq tls
15:14:09 <yoctozepto> looking now
15:14:22 <mnasiadka> There is a change to disable rmq tls testing in CI
15:14:43 <mnasiadka> And a change to requirements to pin kombu and amqp to old versions
15:15:09 <mnasiadka> Because new kombu and amqp tandem causes incompat with oslo.messaging
15:15:33 <yoctozepto> NFV is minor - nobody seems to care
15:15:46 <yoctozepto> escurator was fixed in master I guess, no?
15:15:52 <yoctozepto> "workarounded"
15:15:54 <mgoddard> new issue
15:16:01 <mnasiadka> I saw a failed build today in master...
15:16:07 <yoctozepto> geez
15:16:13 <yoctozepto> so older branches are affected double xD
15:16:27 <yoctozepto> could you add links?
15:16:32 <yoctozepto> I don't see the kombu issue either
15:16:37 <yoctozepto> or is it ovn?
15:17:04 <mnasiadka> No, nova
15:17:18 <mnasiadka> I’ll check and update in 5 mins
15:17:20 <yoctozepto> mnasiadka -> https://etherpad.opendev.org/p/KollaWhiteBoard
15:17:33 <yoctozepto> make sure it always gets there finally :-)
15:18:46 <mgoddard> is anyone working on escurator?
15:19:03 <mgoddard> I have no time this week
15:20:14 <mgoddard> shall we mark it as unbuildable until someone has time to work on it/
15:20:23 <yoctozepto> k-a looks green to me
15:20:38 <yoctozepto> mgoddard: paste a link to failed build in the whiteboard
15:20:47 <mgoddard> there is one
15:20:57 <yoctozepto> that one was fixed
15:21:12 <yoctozepto> you mean it fails exactly the same again?
15:21:23 <yoctozepto> what the escurator
15:21:25 <yoctozepto> :D
15:21:29 <mgoddard> it's slightly different
15:23:15 <mgoddard> last time it was boto
15:23:29 <hrw> kill it with napalm
15:23:39 <mgoddard> curate it
15:23:56 <yoctozepto> we seriously need that tiering approach
15:24:10 <mgoddard> #action mgoddard check if escurator broken and mark unbuildable if so
15:24:12 <yoctozepto> I mean, one practically would not care to have outdated escurator image
15:24:23 <yoctozepto> (for not too long)
15:25:39 <mnasiadka> so, we could switch to binary from packages.elastic.co
15:25:43 <mnasiadka> but there are no aarch64 packages
15:27:40 <mnasiadka> yoctozepto: the Nova issue is in k-a section in whiteboard, but in reality it's Kolla issue, or rather requirements/Nova/oslo.messaging issue :)
15:28:31 <mgoddard> let's move on
15:28:40 <mgoddard> if anyone has time for curator, go for it
15:28:42 <mgoddard> #topci Victoria release planning
15:28:45 <mgoddard> #topic Victoria release planning
15:29:03 <mgoddard> mnasiadka proposed kolla RC1
15:29:10 <mgoddard> what about kolla-ansible and kayobe?
15:29:42 <mgoddard> https://review.opendev.org/#/c/761159/
15:29:42 <patchbot> patch 761159 - releases - Release kolla 11.0.0.0rc1 - 3 patch sets
15:29:50 <mgoddard> kayobe is probably not ready yet
15:30:13 <mnasiadka> kolla-ansible - I would prefer to branch it when simple CI jobs work
15:30:17 <mnasiadka> now it's not the case :)
15:30:27 <mgoddard> #link https://docs.openstack.org/kolla/latest/contributor/release-management.html
15:30:46 <mgoddard> mnasiadka: well, as you said that's a kolla issue :p
15:30:54 <mnasiadka> haha
15:30:55 <mnasiadka> https://launchpad.net/kolla-ansible/+milestone/11.0.0
15:31:05 <mnasiadka> are those bugs stoppers for branching rc?
15:31:44 <mgoddard> only the OVN one is high
15:31:55 <mgoddard> and that's in released code
15:32:14 <mnasiadka> Yeah, but it needs rework - it's on my list, but it doesn't mean it doesn't work at all now :)
15:32:39 <mgoddard> I don't think they are RC blockers
15:32:55 <mnasiadka> ok, I'll raise the k-a change to releases today then
15:33:02 <mnasiadka> what about kayobe?
15:33:58 <mgoddard> https://docs.openstack.org/kayobe/latest/contributor/releases.html
15:34:15 <yoctozepto> mnasiadka: but I don't see k-a periodic jobs failing on nova...
15:35:28 <mnasiadka> yoctozepto: interestingly, do we run {centos8/ubuntu}-source in periodics
15:35:29 <mnasiadka> ?
15:35:33 <mnasiadka> no, just the upgrade ones.
15:36:05 <mnasiadka> and the task that is failing, is waiting for nova-compute to get registered
15:36:28 <mnasiadka> I guess in upgrade it's already registed, so we survive
15:39:16 <mnasiadka> maybe after all we should run those standard tasks in periodic, so we know that they are all right?
15:39:34 <mgoddard> they should be run often enough
15:40:32 <mgoddard> kayobe RC1 depends on kolla RC1
15:40:52 <mgoddard> but if anyone wants to sync the config, that would be good
15:40:56 <mgoddard> and renos
15:41:14 <mnasiadka> I'll look into that tomorrow
15:43:01 <mgoddard> hopefully I'll have more time to push the release tasks forward next week
15:43:08 <mgoddard> #topic Wallaby PTG actions
15:43:19 <mgoddard> #link https://etherpad.opendev.org/p/kolla-wallaby-ptg
15:43:33 <mgoddard> TODO(): High level documentation, eg. examples of networking config, diagrams, justification of use of containers, not k8s etc.
15:43:35 <mgoddard> TODO(): document justification for kolla/kolla-ansible
15:43:37 <mgoddard> TODO(yoctozepto): wait for zuul-runner, try it out, fix issues, document how to use it
15:43:39 <mgoddard> TODO(mgoddard): Poll community for a new kolla klub timeslot
15:43:41 <mgoddard> TODO(Fl1nt): Call for contributor campaign, offer some (limited) mentorship
15:43:43 <mgoddard> TODO(mgoddard): Reach out to existing community members in non EU/US
15:43:45 <mgoddard> TODO() find 'interested parties' for infra image CI work
15:43:47 <mgoddard> TODO(Fl1nt): add a note to documentation (FAQ/troubleshooting?) about new Docker hub limits
15:43:49 <mgoddard> TODO(): ask opendev infra about open source project application process
15:43:51 <mgoddard> TODO(): add tags for register & bootstrap?
15:43:53 <mgoddard> TODO(yoctozepto): make genconfig + deploy-containers work
15:43:55 <mgoddard> TODO(yoctozepto): Deprecate reconfigure command
15:43:57 <mgoddard> TODO(): Modernise the old skool Swift role
15:43:59 <mgoddard> TODO(Fl1nt, or others): PoC and/or spec for podman
15:44:01 <mgoddard> TODO(yoctozepto): work on masakari hostmonitor integration
15:44:03 <mgoddard> TODO(headphoneJames): write up high level description of how Letsencrypt fits together
15:44:05 <mgoddard> TODO(): write a high level design document/spec for kayobe multiple environments
15:44:07 <mgoddard> TODO(): 'Deprecate' devicemapper on stable branches, require some action (set a flag) to override
15:44:09 <mgoddard> TODO(jovial): make a kayobe story for switching to networkmanager
15:44:11 <mgoddard> TODO(dougszu): Strip out Grafana post configure functionality and move it to Kolla-Ansible
15:44:13 <mgoddard> TODO(dougszu): Investigate ansible collections, reference custom playbook repo (e.g. kayobe-ops)
15:44:15 <mgoddard> we have some people assigned
15:44:17 <mgoddard> if there is something you can help with, please add your name on the etherpad
15:44:33 <mgoddard> and if you are assigned to something, please don't forget :)
15:44:55 <mgoddard> any ideas for how we get better at actually doing these?
15:44:57 <mnasiadka> https://review.opendev.org/761433 - kolla-ansible victoria rc1
15:44:58 <patchbot> patch 761433 - releases - Release kolla-ansible 11.0.0.0rc1 - 1 patch set
15:45:26 <mnasiadka> mgoddard: a better tool than etherpad for tracking status? :)
15:45:55 <mgoddard> what could be better?
15:46:55 <yoctozepto> mnasiadka: but we run zun
15:47:03 <yoctozepto> and it's even more demanding
15:47:43 <yoctozepto> https://zuul.openstack.org/build/b0016619a2d846c0bdedb3494ea2bb15
15:48:46 <mgoddard> #topic Open discussion
15:48:54 <mnasiadka> mgoddard: I don't know, I have a perspective that people often vanish for the whole cycle and don't pick up anything they signed up for :)
15:49:05 <mgoddard> that can happen
15:49:13 <mgoddard> also sometimes people appear that weren't at the PTG
15:49:19 <mnasiadka> yoctozepto: do we enable backend tls?
15:49:33 <mgoddard> true, that be the difference
15:49:53 <mnasiadka> mgoddard: I'm just more a graphical board type of a guy, than someone skimming a kilometer long etherpad in search of something I could do :)
15:50:14 <mnasiadka> yes, the bug is related to Nova RMQ backend TLS - nothing else.
15:50:35 <mnasiadka> that's why I proposed a change to disable it in the CI for the time being, and revert when it comes back to shape.
15:50:45 <mgoddard> mnasiadka: when was the last time someone skimmed LP blueprints for something to do (and did it?)
15:51:09 <mnasiadka> yeah, that's another thing, we did some cleanup, but nobody is looking at that either :)
15:51:51 <mnasiadka> Anyway, people signed up for some stuff, let's hope they update etherpad as they go, and we have a blueprint attached to each of those items for easier tracking :)
15:52:00 <mnasiadka> polish complaining mode off
15:52:23 <mgoddard> anything for open discussion?
15:52:38 <yoctozepto> mnasiadka; why Polish?
15:52:46 <yoctozepto> mgoddard, mnasiadka: tls, ack
15:52:50 <mnasiadka> yoctozepto: we are known for complaining all the time
15:52:53 <yoctozepto> need to run periodics with those
15:53:03 <yoctozepto> mnasiadka: pff, I've seen worse ;p
15:53:04 <hrw> I am The Main Complainer at Linaro
15:53:22 <mnasiadka> that's the spirit :)
15:53:30 <hrw> unofficial title for years
15:53:40 <headphoneJames> for lets encrypt integration - are we are initially targeting external vip certs?
15:54:03 <mgoddard> headphoneJames: the API needs to be publicly accessible for ACME to work
15:54:24 <mnasiadka> open discussion - I started moving openvswitch tasks to modules instead of command: - but it needs changes in ansible_sudoers file in kolla_toolbox - https://review.opendev.org/#/c/761403/ - any votes against doing this change?
15:54:25 <patchbot> patch 761403 - kolla - toolbox: Add sudoers for kolla_toolbox become - 1 patch set
15:55:34 <mnasiadka> (not counting that we still have ceph related stuff in ansible_sudoers :-) )
15:56:08 <mnasiadka> yoctozepto: so if ack - then +2 here please https://review.opendev.org/#/c/761282/
15:56:08 <patchbot> patch 761282 - kolla-ansible - CI: Temporarily disable rabbitmq internal tls - 1 patch set
15:56:38 <mgoddard> mnasiadka: find_disks also for swift
15:56:43 <mnasiadka> ah
15:56:46 <mnasiadka> so I'll add that back
15:57:00 <mgoddard> mnasiadka: can we not use a more specific pattern?
15:58:07 <mnasiadka> well, if we add -b flag to ansible run in kolla_toolbox, then most probably we can't, right?
15:58:20 <mnasiadka> unless we run sudo to ansible command, instead of using -b flag?
15:58:59 <mnasiadka> *run ansible command with sudo (sudo ansible bla bla), not the -b (become) function of Ansible
16:00:32 <mnasiadka> mgoddard: unless I'm mistaken in my thinking :)
16:01:08 <mgoddard> you could run ansible with -vvv and see what commands it executes via sudo
16:01:22 <mgoddard> but I guess probably just python <some gross ansible module blob>
16:01:25 <mnasiadka> well, I guess it's python
16:01:49 <mnasiadka> and we only need to run openvswitch_* modules as root, because the permissions on ovs db.sock is root:root
16:02:08 <mnasiadka> unless we add ansible to root group in kolla_toolbox container... hmm
16:02:49 <mgoddard> sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-bjpknhkgjmkrldiquxksmyoydwrudipv ; /usr/bin/python /home/mark/.ansible/tmp/ansible-tmp-1604505753.73-227606553818806/AnsiballZ_openvswitch_db.py
16:03:33 <mnasiadka> that would be one nasty regex with openvswitch_db.py in the end
16:03:35 <mgoddard> what about docker execing as root?
16:04:00 <mnasiadka> well, that's another idea, we could rework kolla_toolbox to exec as root
16:04:40 <mnasiadka> in theory the user: option is for that
16:04:50 <mnasiadka> but I think it didn't work as I expected, will experiment
16:05:09 <mgoddard> as you pointed out, we use the rabbitmq user for rabbitmq
16:05:17 <mgoddard> anyways, meeting time up
16:05:21 <mgoddard> thanks all
16:05:23 <mgoddard> #endmeeting