15:00:31 <mgoddard> #startmeeting kolla
15:00:32 <openstack> Meeting started Wed Nov 25 15:00:31 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:35 <openstack> The meeting name has been set to 'kolla'
15:00:59 <mgoddard> #topic rollcall
15:01:01 <mgoddard> \o
15:01:05 <yoctozepto> o/
15:01:05 <jopdorp> o/
15:01:19 <rafaelweingartne> \o
15:02:15 <mgoddard> #topic agenda
15:02:19 <mgoddard> * Roll-call
15:02:20 <mgoddard> * Announcements
15:02:22 <mgoddard> ** Kolla Wallaby priorities https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:02:24 <mgoddard> ** Stein release now in Extended Maintenance (EM)
15:02:26 <mgoddard> * Review action items from the last meeting
15:02:28 <mgoddard> * CI status
15:02:30 <mgoddard> * Victoria release planning
15:02:32 <mgoddard> * Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:02:34 <mgoddard> * Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:02:36 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:02:36 <mgoddard> * Wallaby PTG actions
15:02:38 <mgoddard> * Review new retirements (Wallaby)
15:02:40 <mgoddard> * Cinder v2 to be dropped in Wallaby http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018697.html
15:02:42 <mgoddard> * Stein Extended Maintenance
15:02:44 <mgoddard> #topic announcements
15:03:06 <mgoddard> #info Kolla Wallaby priorities
15:03:10 <mgoddard> #link Kolla Wallaby priorities https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:03:13 <mgoddard> #undo
15:03:14 <openstack> Removing item from minutes: #link https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:03:19 <mgoddard> #link https://etherpad.opendev.org/p/kolla-wallaby-priorities
15:03:32 <mgoddard> Voting now finished, we have our priorities
15:03:38 <mgoddard> I've added them to the whiteboard
15:03:51 <mgoddard> #link https://etherpad.opendev.org/p/KollaWhiteBoard
15:04:51 <mgoddard> Feel free to add yourself as an owner/interested party on one of the priorities
15:04:54 <mgoddard> #info Stein release now in Extended Maintenance (EM)
15:05:05 <mgoddard> Final releases have been created
15:05:19 <mgoddard> We can discuss more later
15:05:22 <mgoddard> Any others?
15:06:43 <mgoddard> #topic Review action items from the last meeting
15:06:58 <mgoddard> mgoddard to email openstack-discuss about final reminder for wallaby priority voting
15:07:00 <mgoddard> yoctozepto fix NFV
15:07:05 <mgoddard> mgoddard did his
15:07:15 <mgoddard> yoctozepto seems to have been busy
15:07:26 <mgoddard> do you want another action?
15:08:23 <yoctozepto> mgoddard: yes, please
15:08:29 <yoctozepto> mgoddard: health issues
15:08:53 <mgoddard> yoctozepto: sorry to hear
15:09:03 <mgoddard> #action yoctozepto fix NFV
15:09:13 <mgoddard> #topic CI status
15:09:34 <mgoddard> 
15:09:48 <mgoddard> docker rate limits we can discuss again later
15:10:22 <mgoddard> https://bugs.launchpad.net/nova/+bug/1902696
15:10:23 <openstack> Launchpad bug 1902696 in oslo.messaging "nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'" [Undecided,New]
15:10:36 <mgoddard> there was another patch for requirements, but it didn't help us
15:11:16 <mgoddard> I'm sure there was a version of the patch that worked, do we need to go back and find out what it was
15:11:36 <yoctozepto> 5.0.2 does not help
15:12:15 <mgoddard> no
15:13:16 <mgoddard> looks like PS1 passed
15:13:26 <mgoddard> nov 7 11:35PM
15:14:03 <yoctozepto> PS1 of which one?
15:14:04 <mgoddard> at that time, the requirements patch was PS2: https://review.opendev.org/c/openstack/requirements/+/761194/2
15:14:08 <yoctozepto> OK
15:14:39 <mgoddard> which pinned amqp<=2.6.1
15:14:40 <yoctozepto> so yeah, it simply says new kombu is b0rken
15:14:59 <mgoddard> and kombu<=5.0.1
15:15:37 <mgoddard> current patch allows amqp 5.0.2
15:16:10 <yoctozepto> and kombu 5.0.2
15:16:41 <mgoddard> yes
15:17:50 <yoctozepto> duh, new gerrit clumsy
15:17:55 <mgoddard> yeah, and slow
15:18:05 <mgoddard> hopefully it'll improve
15:18:30 <yoctozepto> ++
15:18:33 <mgoddard> #action mgoddard to try reverting to https://review.opendev.org/c/openstack/requirements/+/761194/2
15:19:11 <mgoddard> I think other CI issues are unchanged
15:19:29 <mgoddard> #topic Victoria release planning
15:19:43 <mgoddard> https://etherpad.opendev.org/p/KollaWhiteBoard
15:19:49 <mgoddard> L144 lists release blockers
15:20:11 <mgoddard> Kolla has the rabbitmq TLS issue we were just discussing
15:20:20 <yoctozepto> anyone looked into a/a?
15:20:24 <mgoddard> Kolla ansible has cinder-volume active/active
15:20:40 <mgoddard> I don't think so
15:20:59 <yoctozepto> heh
15:21:15 <mgoddard> not beyond mnasiadka's patch anyway
15:21:36 <mgoddard> I'll try to put some time into it tomorrow
15:21:41 <yoctozepto> yeah
15:21:47 <yoctozepto> best approach with CI testing
15:21:56 <yoctozepto> what happens if we upgrade to new approach
15:22:01 <yoctozepto> when we have volumes in place
15:22:07 <yoctozepto> might be worth to have this in general
15:22:13 <yoctozepto> instead of cleaning up
15:23:29 <mgoddard> yes
15:24:02 <mgoddard> create, test, delete, create, test, upgrade, test, delete, create, test, delete
15:24:13 <mgoddard> something like that
15:24:15 <yoctozepto> ++
15:24:37 <yoctozepto> for simplicity
15:24:50 <yoctozepto> we can treat reconf = upgrade
15:24:55 <yoctozepto> would not hurt
15:25:58 <mnasiadka> well, we need to be sure that no volumes are in -ing state before upgrade
15:26:03 <mgoddard> although it's effectively a noop, so we'd be unlikely to catch anything
15:26:25 <yoctozepto> (ignore the above, we don't test after reconf)
15:26:28 <mgoddard> (it = reconfigure in CI)
15:26:33 <mnasiadka> (and remove old agents from db or ask the user to remove them)
15:26:37 <mnasiadka> after upgrade
15:26:54 <mnasiadka> or am I missing something major? :)
15:27:08 <yoctozepto> we are discussing testing to see the impact
15:27:18 <yoctozepto> and strenghten our testing in general
15:27:32 <yoctozepto> volumes disappearing due to an upgrade is a big no-no
15:28:11 <mgoddard> ok, let's move on
15:28:21 <mgoddard> #topic Dockerhub pull rate limits https://etherpad.opendev.org/p/docker-pull-limits
15:29:05 <mgoddard> I was thinking, given the lack of any action on this, how about we go with hrw's suggestion to publish less frequently?
15:29:29 <mgoddard> it should help, although not guaranteed
15:30:12 <mgoddard> pretty simple to implement though, and it might be nice to have a pattern for less frequent publishing, e.g. for EM branches
15:30:32 <mgoddard> any thoughts?
15:30:46 <yoctozepto> but we get penalized for *pulling*
15:30:51 <yoctozepto> not pushing
15:30:55 <mgoddard> yes
15:31:10 <mgoddard> but the less we push, the less we have to pull into registry mirrors
15:31:19 <mgoddard> since they should cache images
15:31:21 <rafaelweingartne> Would it be possible for the community/foundation to implement a proxy-registry somehow?
15:31:24 <yoctozepto> hmm, clever
15:31:33 <wuchunyang> i think reconfigure can do some reload work(or restart one by one without downtime)
15:32:07 <mgoddard> wuchunyang: rate limits are per 6 hours, not per second :)
15:32:46 <mgoddard> rafaelweingartne: opendev infra provide registry mirrors, which we use
15:32:46 <yoctozepto> weekly images sound fine
15:32:51 <wuchunyang> ok..
15:32:56 <yoctozepto> and absurdly easy to implement now
15:33:03 <yoctozepto> just switch the pipeline
15:33:05 <yoctozepto> :D
15:33:30 <rafaelweingartne> mgoddard: so, we do not actually need to worry about the pull problem, right? If we document on how people can use these mirrors then
15:33:37 <mgoddard> oh nice, periodic-weekly exists
15:33:44 <yoctozepto> indeed
15:33:54 <yoctozepto> the only problem is
15:33:59 <yoctozepto> the failures
15:33:59 <yoctozepto> :D
15:34:06 <yoctozepto> ever-b0rken images
15:34:12 <yoctozepto> never-new images
15:34:20 <yoctozepto> sad reality
15:34:53 <mgoddard> rafaelweingartne: we're mostly focussing on how it affects CI here. Users need their own solution, such as a local registry or registry mirror
15:35:17 <rafaelweingartne> I see, that is actually what we are using.
15:35:28 <mgoddard> rafaelweingartne: sensible, even before the rate limit
15:35:41 <mgoddard> yoctozepto: true. I was thinking more like every 2-3 days
15:35:56 <mgoddard> with some early exit in the publish job
15:36:50 <mgoddard> if hash_of_job_info % 7 in days_to_publish then publish
15:37:33 <mgoddard> would be nice if zuul gave the date of the last successful run of the job
15:37:47 <mgoddard> possibly we could query it
15:38:02 <yoctozepto> sounds sensible to query
15:38:04 <mgoddard> that would allow for failures
15:38:09 <yoctozepto> though never used the zuul api before
15:38:17 <mgoddard> it's easy enough
15:38:18 <yoctozepto> mhm
15:39:06 <mgoddard> anyway, some solution in there
15:39:09 <mgoddard> anyone want to pick it up?
15:39:22 <mgoddard> I suppose periodic-weekly would be an easy win
15:39:40 <mgoddard> although they probably all run on a sunday
15:39:48 <yoctozepto> and an easy self-sabotage ;d
15:40:08 <yoctozepto> let's go with the zuul querying
15:40:21 <yoctozepto> though it's all problematic as these jobs only ever run periodically
15:40:28 <yoctozepto> need to simulate in DNM first
15:40:45 <mgoddard> yes
15:41:08 <mgoddard> I think a static hash makes sense. It distributes jobs over days
15:41:31 <mgoddard> a system failure would lead to stampeding if we use the last success
15:41:55 <yoctozepto> so you prefer stochastic approaches
15:42:01 <yoctozepto> fine by me
15:42:36 <mgoddard> I don't mind too much, as long as it gets fixed :)
15:43:03 <yoctozepto> somebody fix :-)
15:43:09 <mgoddard> it's available, if someone wants to pick it up
15:43:13 <mgoddard> #topic Cinder active/active https://bugs.launchpad.net/kolla-ansible/+bug/1904062
15:43:15 <openstack> Launchpad bug 1904062 in kolla-ansible wallaby "external ceph cinder volume config breaks volumes on ussuri upgrade" [High,In progress] - Assigned to Michal Nasiadka (mnasiadka)
15:43:35 <mgoddard> I don't think we have too much to discuss on this one, other than we need to get on with it
15:44:12 <mgoddard> #topic Wallaby PTG actions
15:44:28 <mgoddard> TODO(): High level documentation, eg. examples of networking config, diagrams, justification of use of containers, not k8s etc.
15:44:30 <mgoddard> TODO(): document justification for kolla/kolla-ansible
15:44:32 <mgoddard> TODO(yoctozepto): wait for zuul-runner, try it out, fix issues, document how to use it
15:44:34 <mgoddard> TODO(mgoddard): Poll community for a new kolla klub timeslot
15:44:36 <mgoddard> TODO(Fl1nt): Call for contributor campaign, offer some (limited) mentorship
15:44:38 <mgoddard> TODO(mgoddard): Reach out to existing community members in non EU/US
15:44:40 <mgoddard> TODO() find 'interested parties' for infra image CI work
15:44:42 <mgoddard> TODO(Fl1nt): add a note to documentation (FAQ/troubleshooting?) about new Docker hub limits
15:44:44 <mgoddard> TODO(): ask opendev infra about open source project application process
15:44:46 <mgoddard> TODO(): add tags for register & bootstrap?
15:44:48 <mgoddard> TODO(yoctozepto): make genconfig + deploy-containers work
15:44:50 <mgoddard> TODO(yoctozepto): Deprecate reconfigure command
15:44:52 <mgoddard> TODO(): Modernise the old skool Swift role
15:44:54 <mgoddard> TODO(Fl1nt, or others): PoC and/or spec for podman
15:44:56 <mgoddard> TODO(yoctozepto): work on masakari hostmonitor integration
15:44:58 <mgoddard> TODO(headphoneJames): write up high level description of how Letsencrypt fits together
15:45:00 <mgoddard> TODO(): write a high level design document/spec for kayobe multiple environments
15:45:02 <mgoddard> TODO(): 'Deprecate' devicemapper on stable branches, require some action (set a flag) to override
15:45:04 <mgoddard> TODO(jovial): make a kayobe story for switching to networkmanager
15:45:06 <mgoddard> TODO(dougszu): Strip out Grafana post configure functionality and move it to Kolla-Ansible
15:45:08 <mgoddard> TODO(dougszu): Investigate ansible collections, reference custom playbook repo (e.g. kayobe-ops)
15:45:10 <mgoddard> any updates?
15:45:12 <mgoddard> anyone want to pick one up that is not assigned?
15:45:37 <headphoneJames> the high level description: https://etherpad.opendev.org/p/kolla-ansible-letsencrypt-https
15:45:42 <rafaelweingartne> it is not in this list, but we did update https://review.opendev.org/c/openstack/kolla-ansible/+/695432
15:45:44 <yoctozepto> that was quick
15:45:55 <rafaelweingartne> I added the requested documentation on how to create a DEV env. for testing
15:47:17 <mgoddard> thanks rafaelweingartne, that should help reviewers & testers
15:47:42 <rafaelweingartne> Cool, we are guessing that now, people would be more confident in testing
15:47:46 <rafaelweingartne> and then we can move on with that
15:48:20 <mgoddard> yes, I think we should prioritise it once Victoria is released
15:48:54 <rafaelweingartne> awesome, thanks!
15:49:05 <mgoddard> headphoneJames: a little off topic, but have you been communicating with Jason?
15:49:17 <wuchunyang> Deprecate reconfigure command   this task i can help
15:49:22 <headphoneJames> I did reach out to him
15:49:54 <headphoneJames> he knows that I'm running with LE
15:49:57 <mgoddard> ok, great
15:50:07 <headphoneJames> I also let him know that I updated the spec
15:50:14 <mgoddard> as long as we're keeping him in the loop
15:50:25 <wuchunyang> anyone uses podman in production ?
15:51:02 <wuchunyang> i use podman to deploy ceph in production, but i hit a podman bug..
15:51:26 <mgoddard> wuchunyang: the list is on https://etherpad.opendev.org/p/kolla-wallaby-ptg, feel free to add your nick
15:51:38 <wuchunyang> ok
15:52:13 <mgoddard> wuchunyang: I think we normally use docker
15:52:20 <mgoddard> #topic Review new retirements (Wallaby)
15:52:25 <mgoddard> yoctozepto: is this you?
15:52:38 <wuchunyang> yes, i think docker is more reliable ..
15:52:44 <yoctozepto> yees
15:53:01 <dswebb> wuchunyang, we use a mix (docker for openstack, podman for ceph)
15:53:03 <yoctozepto> we got a bit of retirements
15:53:10 <yoctozepto> on the mailing list
15:53:22 <mgoddard> should we allow time for people to come forward to help?
15:53:23 <yoctozepto> so we might want to deprecate as well
15:53:27 <wuchunyang> dswebb me too.. but our ceph hit a podman bug
15:53:39 <yoctozepto> yeah, I'm fine with allowing more time
15:53:47 <mgoddard> I guess deprecation doesn't hurt
15:53:48 <dswebb> you having issues with the node exporter not starting properly on deploy?
15:53:48 <yoctozepto> just keep this in the back of our minds
15:53:58 <mgoddard> good thinking
15:54:02 <yoctozepto> you can action me on it
15:54:07 <yoctozepto> I will propose notes
15:54:10 <yoctozepto> to keep around
15:54:19 <mgoddard> #action yoctozepto deprecate the retired Wallabies
15:54:26 <yoctozepto> thx
15:54:30 <wuchunyang> dswebb    https://github.com/containers/podman/issues/2553
15:54:34 <mgoddard> #topic Cinder v2 to be dropped in Wallaby http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018697.html
15:54:47 <mgoddard> #link http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018697.html
15:54:50 <mgoddard> yoctozepto again?
15:55:05 <yoctozepto> you guessed it!
15:55:12 <mgoddard> I actually missed that one
15:55:28 <yoctozepto> now, it's trickier than the previous one
15:55:39 <yoctozepto> we need to handle the drop and removal of existing
15:55:51 <yoctozepto> it should not be breaking if we forget
15:55:57 <yoctozepto> but better play it safe
15:56:09 <mgoddard> we need a way to track things like this
15:56:24 <yoctozepto> another etherpad?
15:56:29 <yoctozepto> whiteboard feels lengthy
15:56:33 <yoctozepto> or maybe
15:56:34 <mgoddard> was thinking another list next to priorities
15:56:36 <yoctozepto> just use wiki
15:56:45 <yoctozepto> oh well
15:56:46 <mgoddard> considering it only has one item currently :)
15:56:52 <yoctozepto> whatever works
15:56:56 <yoctozepto> true that
15:57:04 <mgoddard> etherpads get forgotten
15:57:24 <mnasiadka> we can always start in a contest for longest etherpad :)
15:57:58 <yoctozepto> mnasiadka: ++
15:58:06 <mnasiadka> yoctozepto: I remember we had the discussion before, that we need a role/change in service-ks-register to remove endpoints
15:58:12 <yoctozepto> hopefully the prizes are worth it
15:58:19 <yoctozepto> mnasiadka: we had
15:58:26 <yoctozepto> but there was no urgency
15:58:29 <yoctozepto> now there is
16:00:09 <mgoddard> I've added it as a priority for kolla-ansible
16:00:11 <hrw> what about one patch to deprecate and next one to remove with WIP status?
16:00:30 <hrw> or similar
16:00:44 <mgoddard> hrw: for cinder or retired projects?
16:00:50 <hrw> retired
16:00:53 <mgoddard> ok
16:01:04 <mgoddard> can do, was just hoping to avoid possibly unnecessary work
16:01:13 <mgoddard> if someone picks up the projects
16:01:35 <mgoddard> but I won't/can't stop anyone proposing a patch
16:01:40 <mgoddard> #topic Stein Extended Maintenance
16:01:45 <mgoddard> last topic
16:01:48 <mgoddard> Stein is now EM
16:02:27 <hrw> so we can stop publish/care?
16:02:33 <mgoddard> I was hoping to see something on https://docs.openstack.org/kolla/latest/contributor/release-management.html about what we normally do for EM
16:02:37 <mgoddard> but there is nothing
16:02:48 <mgoddard> I think, we do this
16:03:11 <mgoddard> stop backports by default, but accept if proposed
16:03:37 <mgoddard> stop publishing (although not usually immediately)
16:04:02 <mgoddard> we could also try switching to stable branches in source images (as we did in ussuri)
16:04:11 <mgoddard> and switch to weekly publish
16:04:14 <mgoddard> thoughts?
16:04:58 <mnasiadka> well, switching to stable branches could be good for users using Stein
16:05:00 <hrw> it is em. I would do one final release if there were changes since previous
16:05:13 <mgoddard> final release is done
16:05:59 <hrw> I would end.
16:06:07 <hrw> do final publish and done
16:06:20 <hrw> switching to stable branches can bring new issues
16:07:18 <mgoddard> rocky was last published 8 months ago
16:07:22 <hrw> when someone propose backport with sensible reason then we merge and let users do build on they own or wait for weekly/monthly publish
16:07:45 <hrw> mgoddard: and no one asked about rocky so we do not recognize that name anymore?
16:08:17 <mgoddard> ok, we're past time
16:08:17 <mgoddard> thanks all
16:08:19 <mgoddard> #endmeeting