15:00:16 <mgoddard> #startmeeting kolla
15:00:16 <openstack> Meeting started Wed Feb 19 15:00:16 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:19 <openstack> The meeting name has been set to 'kolla'
15:00:19 <mgoddard> #topic rollcall
15:00:21 <mgoddard> \o
15:00:21 <cosmicsound> o/
15:00:37 <osmanlicilegi> o/
15:00:52 <dougsz> \o
15:01:03 <osmanlicilegi> o/
15:01:06 <yoctozepto> o/
15:02:17 <mgoddard> #topic announcements
15:02:31 <mgoddard> #undo
15:02:32 <openstack> Removing item from minutes: #topic announcements
15:02:34 <mgoddard> #topic agenda
15:02:40 <mgoddard> * Roll-call
15:02:42 <mgoddard> * Announcements
15:02:44 <mgoddard> * Review action items from last meeting
15:02:46 <mgoddard> * CI status
15:02:48 <mgoddard> * Ussuri release planning (kolla & kolla ansible)
15:02:50 <mgoddard> * Ussuri release planning (kayobe)
15:02:52 <mgoddard> ** XenAPI deprecated in Nova Train, removal pending
15:02:54 <mgoddard> * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig
15:02:56 <mgoddard> #topic announcements
15:03:03 <mgoddard> Looks like we have none on the agenda. Anyone have any?
15:03:11 <osmanlicilegi> nope
15:04:54 <dougsz> I was going to raise an issue, but I think I have a workaround.
15:05:06 <dougsz> so, nope, for now at least
15:05:13 <mgoddard> We can discuss in open discussion if you like
15:05:20 <mgoddard> #topic review action items from last meeting
15:05:32 <mgoddard> yoctozepto to deprecate mongodb for ubuntu, drop for centos
15:05:46 <mgoddard> I think I saw it proposed.
15:06:01 <mgoddard> good work
15:06:04 <mgoddard> #topic CI status
15:06:31 <mgoddard> "general issues with new py, affects tripleo as well, and jobs used from k-a"
15:06:40 <mgoddard> what is this?
15:06:43 <mgoddard> still an issue?
15:08:19 <mgoddard> anyone?
15:08:36 <osmanlicilegi> I'm not sure about the details
15:08:36 <mgoddard> kolla CI looks green
15:09:12 <mgoddard> date is 13/2/2020, possibly an old one
15:09:20 <yoctozepto> we got past that
15:09:25 <mgoddard> cool
15:09:30 <mnasiadka> yeah, but still the workaround is applied I guess
15:09:36 <mnasiadka> with installing newer virtualenv or something
15:09:37 <yoctozepto> it is
15:09:43 <yoctozepto> testing revert now
15:09:45 <mgoddard> oh it's that thing
15:09:58 <yoctozepto> looks passy
15:10:01 <yoctozepto> https://review.opendev.org/#/c/707623/
15:10:01 <patchbot> patch 707623 - kolla - Revert "Upgrade virtualenv in pre" - 1 patch set
15:10:19 <mnasiadka> yoctozepto: kolla-ansible part as well? :)
15:10:35 <yoctozepto> well, you won't see from here unless you have zuul plugin in browser
15:10:36 <yoctozepto> yeah
15:10:38 <mgoddard> ok, let's try to get those merged
15:10:47 <mgoddard> onto k-a
15:10:52 <mgoddard> stein ubuntu binary fails on neutron migrations (foreign key blah blah)
15:11:03 <yoctozepto> it failed recently yeah
15:11:11 <mgoddard> I remember seeing it
15:11:17 <mgoddard> is it a persistent failure?
15:11:28 <yoctozepto> let's check :-)
15:11:53 <yoctozepto> https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-ubuntu-binary&branch=stable%2Fstein
15:12:17 <mgoddard> looks bad
15:12:25 <mgoddard> anyone want to investigate?
15:12:25 <mnasiadka> neutron backported the fail to stein? :D
15:12:52 <yoctozepto> rechecking https://review.opendev.org/#/c/706078/
15:12:53 <patchbot> patch 706078 - kolla-ansible (stable/stein) - Fix multiple issues with MariaDB handling - 1 patch set
15:12:57 <yoctozepto> mnasiadka: no idea
15:13:04 <yoctozepto> mnasiadka: somehow only binary
15:13:07 <mgoddard> I wonder why it's binary only. Maybe the package needs a bump?
15:13:13 <mnasiadka> I already raised two bugs to neutron, not gonna look into third one :)
15:13:15 <mgoddard> or stein needs a release
15:13:23 <yoctozepto> in uca
15:13:26 <mgoddard> yes
15:13:38 <yoctozepto> but they surely should know theu have b0rken debs
15:13:58 <yoctozepto> let's se
15:14:01 <mnasiadka> we could check the neutron version in UCA stein-testing - if there's a version difference
15:14:31 <mgoddard> sounds like a volunteer
15:15:03 <mgoddard> #action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary
15:15:07 <mgoddard> thanks :)
15:15:28 <mgoddard> report back if it's more complicated than that
15:15:42 <openstackgerrit> Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/707131
15:15:59 <mgoddard> "general issues with new py when run from kolla side"
15:16:03 <openstackgerrit> Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api  https://review.opendev.org/707131
15:16:08 <mgoddard> assume this is the virtualenv issue?
15:16:27 <yoctozepto> mgoddard: the same
15:16:33 <mgoddard> removing
15:16:39 <mnasiadka> mgoddard: 14.04 got to updates some time ago, probably it's broken-ish
15:16:51 <mgoddard> 14.04??
15:16:59 <mnasiadka> neutron-server 14.04
15:17:05 <mnasiadka> whatever it is on Ubuntu side :)
15:17:10 <mgoddard> lol, thought you meant ubuntu 14.04
15:17:12 <mnasiadka> http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html
15:17:36 <yoctozepto> mnasiadka: and what is current
15:18:06 <mgoddard> 14.1.0
15:18:18 <mnasiadka> I don't like the 1 in the middle :)
15:18:24 <yoctozepto> 14.0.4 vs 14.1.0
15:18:27 <yoctozepto> mnasiadka: why?
15:18:44 <yoctozepto> it's a bumpy in reqs most likely
15:18:46 <mnasiadka> it basically means normally it's a big change or some feature backport :)
15:19:05 <yoctozepto> or exactly fix for our issue which required dep update ;-)
15:19:19 <yoctozepto> anyways, mnasiadka pokes uca and we are good
15:19:20 <mnasiadka> might be
15:19:23 <mgoddard> #undo
15:19:24 <openstack> Removing item from minutes: #link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html
15:19:28 <mnasiadka> well, another bug to raise I guess
15:19:30 <mgoddard> #undo
15:19:32 <openstack> Removing item from minutes: #action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary
15:19:36 <mgoddard> #link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html
15:19:52 <mgoddard> #action mnasiadka request neutron 14.1.0 in stein UCA
15:20:01 <yoctozepto> pretty precise
15:20:13 <mgoddard> precise pangolin?
15:20:24 <mgoddard> anyways
15:20:39 <yoctozepto> xD
15:20:41 <mgoddard> #topic Ussuri release planning (kolla & kolla ansible)
15:21:06 <mgoddard> Any Ussuri features we need to discuss this week?
15:21:52 <mnasiadka> maybe not a feature, but a community goal - can I get some reviews on https://review.opendev.org/#/c/707800/ ?
15:21:52 <patchbot> patch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets
15:22:24 <mgoddard> will try to get a looko
15:22:31 <mnasiadka> good
15:22:33 <yoctozepto> "So You Want to Contribute..." is in the goal? ;D
15:22:45 <mnasiadka> would love some reviews on OVN patch
15:22:57 <mgoddard> my review queue seems to be growing faster than I can process
15:22:59 <mnasiadka> yoctozepto: complain to the goal coordinator :)
15:23:09 <mgoddard> mnasiadka: I started a review of OVN :)
15:23:31 <mnasiadka> mgoddard: good to hear :)
15:23:35 <osmanlicilegi> ah I feel ashame now. mnasiadka I'm so sorry I couldn't spent time for the comments you wait from me :|
15:23:40 <mgoddard> will try to finish
15:24:22 <mnasiadka> osmanlicilegi: easy, fix your datacenter :)
15:24:27 <mgoddard> can we mark https://blueprints.launchpad.net/kolla-ansible/+spec/ceph-ansible complete?
15:24:51 <mnasiadka> well, I'd like to add prechecks for keys existence - but it's rather optional and nice to have
15:24:58 <mgoddard> and https://blueprints.launchpad.net/kolla-ansible/+spec/remove-ceph
15:25:10 <mnasiadka> yeah, that one for sure
15:25:15 <mgoddard> mnasiadka: the blueprint is just about CI
15:25:36 <mnasiadka> mgoddard: right, I just took the extra mile to make it more user friendly :)
15:25:57 <mgoddard> always appreciated
15:25:59 <mnasiadka> whoa, remove-ceph was never approved :)
15:26:20 <yoctozepto> mnasiadka: WHAT HAVE YOU DONE?!
15:26:55 <mnasiadka> yoctozepto: well, it's better to beg for forgiveness, than to ask for permission - that's my life motto :)
15:27:09 <yoctozepto> mnasiadka: gets things done faster
15:27:12 <mgoddard> rm -rf; echo sorry
15:27:27 <mnasiadka> yeah, sorry is the key
15:27:28 <yoctozepto> mgoddard: I've almost spilled my drink
15:27:48 <mgoddard> apologies
15:28:12 <mgoddard> although I did already say sorry
15:29:19 <mgoddard> anything else to discuss?
15:29:39 <yoctozepto> only stuff for later
15:29:52 <mgoddard> #topic Ussuri release planning (kayobe)
15:30:02 <mgoddard> dougsz
15:30:10 <mgoddard> Can't see priteau or jovial
15:30:56 <yoctozepto> where them folks at
15:30:59 <yoctozepto> where, where
15:31:19 <dougsz> hmm.. what do we have, possibly a cells config cleanup
15:32:04 <mgoddard> dougsz: what needs to be done for that?
15:32:18 <cosmicsound> So ceph stays? :D
15:32:29 * cosmicsound chuckels
15:32:41 <mgoddard> cosmicsound: ceph? too late - it's gone
15:32:43 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds  https://review.opendev.org/707818
15:32:51 <mgoddard> aha, priteau & jovial[m]
15:32:58 <priteau> o/
15:33:07 <jovial[m]> hi
15:33:12 <dougsz> mgoddard: Anything to make it more manageable for high numbers of cells
15:33:50 <dougsz> on the topic of ceph.. do we have any Kayobe references to remove?
15:34:07 <mgoddard> there is more work cells planned for k-a, will you be looking at it before ussuri?
15:34:38 <dougsz> We need shared cell controllers sooner or later
15:34:45 <dougsz> I will try to
15:34:52 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs  https://review.opendev.org/708318
15:35:00 <dougsz> We already have some half backed patches around that
15:35:07 <mgoddard> yep
15:35:15 <dougsz> *baked
15:35:27 <mgoddard> I think a separate nova-compute role might make it more scalable, but that needs investigation
15:35:41 <dougsz> agreed
15:35:49 <mgoddard> on the ceph removal, I created https://storyboard.openstack.org/#!/story/2007295
15:36:02 <jovial[m]> a high priority for me would be to implement the custom extension points proposal
15:36:03 <mgoddard> we have a role to label block devices
15:36:17 <mgoddard> anyone want to pick up ceph removal?
15:36:31 <mgoddard> should be therapeutic if you like removing code
15:36:52 <jovial[m]> I will volunteer if no one else wants it
15:37:22 <mgoddard> #action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295
15:37:23 <mgoddard> thanks
15:37:25 <dougsz> thanks jovial[m]
15:37:26 <yoctozepto> mgoddard: is it possible to volunteer from outside?
15:37:30 <yoctozepto> ah, too late :-)
15:37:41 <mgoddard> anyone can volunteer to work on kayobe
15:37:50 <mgoddard> there is no outside in open source :)
15:37:55 <yoctozepto> well, you know, making it official
15:37:57 <yoctozepto> and stuff
15:38:16 <yoctozepto> could get to know kayobe better
15:38:38 <mgoddard> you want to take it?
15:38:47 <mgoddard> I'm sure jovial[m] won't mind too much :)
15:38:53 <yoctozepto> mgoddard: yeah, therapeutic
15:38:59 <mgoddard> #undo
15:39:00 <openstack> Removing item from minutes: #action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295
15:39:09 <jovial[m]> fine by me :)
15:39:10 <mgoddard> #action yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295
15:39:13 <mgoddard> thanks yoctozepto
15:39:19 <dougsz> +1
15:39:19 <yoctozepto> ;-)
15:39:37 <mgoddard> jovial[m] is away for a while anyway
15:40:09 <mgoddard> jovial[m]: you would like to see the custom extension points proposal
15:40:30 <mgoddard> https://storyboard.openstack.org/#!/story/2001663
15:40:30 <cosmicsound> :)
15:40:32 <dougsz> We certainly have an issue with exporters, would be nice to use it for that
15:40:46 <mgoddard> jovial[m]: want to pick it up?
15:41:15 <jovial[m]> for sure -  lots of people want this I believe
15:41:25 <mgoddard> +1
15:41:47 <mgoddard> it will make for a smoother ride in many cases
15:42:00 <priteau> Extension points and multiple environments are two big requested features
15:42:02 <mgoddard> #action jovial[m] to work on custom extension points
15:42:05 <mgoddard> yep
15:42:14 <yoctozepto> folks, remember some extensions might better fit in k-a
15:42:18 <yoctozepto> and these are welcome as well
15:42:22 <yoctozepto> mgoddard to correct me :-)
15:42:33 <mgoddard> yoctozepto: you are right
15:42:46 <mgoddard> we will try to use judgement
15:42:55 <yoctozepto> OK
15:43:15 <mgoddard> this is about a framework for running hook playbooks at various points in kayobe workflows
15:43:49 <yoctozepto> mgoddard: yeah, skimmed over story
15:44:01 <mgoddard> onto the other big feature then - multiple environments
15:44:13 <mgoddard> https://storyboard.openstack.org/#!/story/2002009
15:44:21 <mgoddard> anyone interested in picking this up?
15:44:30 <mgoddard> it's a bigger task
15:44:39 <dougsz> not enough cycles, sorry
15:44:42 <mgoddard> could potentially be a collaboration between multiple people
15:45:32 <dougsz> Maybe someone is willing to help fund it?
15:45:33 <mgoddard> I'd like to be involved to some extent, but CentOS 8 is eating my cycles at the moment
15:46:03 <jovial[m]> another must have feature in my opinion
15:46:15 <mgoddard> I think we have a funder (getting a bit internal here though :))
15:46:31 <mgoddard> ok, no takers yet
15:46:35 <mgoddard> keep it in mind
15:46:38 <yoctozepto> mgoddard: slightly off topic; bored with bifrost, how is bikolla going? I really liked the "containerized bifrost"
15:46:40 <jovial[m]> how long have we got until the next release?
15:47:08 <mgoddard> Kolla feature freeze: Apr 27 - May 01
15:47:08 <yoctozepto> mgoddard: has it any roadmap or whatever?
15:47:10 <mgoddard> Kolla RC1: May 11 - May 15
15:47:12 <mgoddard> Kolla final release: Jun 1 - Jun 5
15:47:16 <yoctozepto> can we fit it any place?
15:47:38 <mgoddard> when you say containerised bifrost you don't mean our current bifrost container :)
15:47:50 <yoctozepto> mgoddard: I mean bikolla hence ""
15:47:53 <yoctozepto> :D
15:48:08 <mgoddard> dougsz was looking at it recently
15:48:19 <yoctozepto> dougsz: how's bikolla? :-)
15:48:29 <mgoddard> still at the proposal/design stage though IIRC
15:48:47 <mgoddard> or possibly just blocked on cycles :)
15:48:52 <yoctozepto> I'd like to be in for any eventual discussion
15:48:53 <dougsz> we need it for deploying a large number of hypervisors
15:49:20 <dougsz> I was also wondering about just using the top level ironic for that
15:49:28 <mgoddard> noted yoctozepto
15:49:36 <yoctozepto> thanks
15:49:52 <yoctozepto> dougsz: what do you mean by "top level"
15:50:04 <dougsz> the overcloud ironic, not bifrost ironic
15:50:08 <mgoddard> that is the other option
15:50:32 <mnasiadka> dougsz: don't you want to use it for standing up the overcloud? :)
15:50:45 <yoctozepto> yeah, I'm confused too
15:51:00 <dougsz> well, bifrost can still handle a small number of controllers, just not the hypervisors
15:51:02 <yoctozepto> I thought bikolla is about deploying standalone ironic with kolla
15:51:12 <yoctozepto> so bifrost via kolla without bifrost
15:51:28 <mgoddard> correct
15:51:31 <mnasiadka> dougsz: looking from Kolla side - I would be happy to deprecate and remove bifrost container, and this way we would need to maintain it :)
15:52:12 <yoctozepto> +1
15:52:20 <dougsz> yeah, it is balancing the maintenance of bifrost against the work required for bikolla
15:52:21 <jovial[m]> would kayobe need a name change? ;-)
15:52:45 <yoctozepto> jovial[m]: not it's a snake
15:52:51 <yoctozepto> be become bikolla
15:53:04 <mgoddard> kayobikolla?
15:53:05 <yoctozepto> kolla on bi...kolla on bi..kolla .....
15:53:16 <yoctozepto> perpetum kayobile
15:53:23 <jovial[m]> :D
15:53:24 <mnasiadka> mgoddard: looking from another perspective - maybe it would make sense to document bikolla in k-a and make it easier to use for users? I don't think anybody is really aware of using bikolla instead of bifrost.
15:53:40 <yoctozepto> ^ that's also what I meant
15:53:48 <yoctozepto> mnasiadka better at putting my thoughts in words
15:53:52 <yoctozepto> thanks mnasiadka
15:54:05 <mgoddard> we could do that
15:54:23 <yoctozepto> "but we won't"
15:54:36 <mnasiadka> well, anybody with spare cycles can do that
15:54:42 <yoctozepto> mnasiadka to pick it up?
15:54:54 <yoctozepto> I am too nooby atm ;p
15:54:56 <mnasiadka> I can add it to my list, although not with a very high priority :)
15:55:26 <yoctozepto> high is fine ;-)
15:55:29 <mnasiadka> hell, we could even do deploy-bikolla with some stock inventory file and stock globals to deploy it on localhost
15:55:39 <mgoddard> docs to deploy the containers should be easy enough
15:56:12 <mgoddard> using the thing and a full tested procedure would take longer
15:56:41 <yoctozepto> that said, does kayobe have docs on how ci is configured?
15:56:44 <yoctozepto> aka what is tested
15:56:52 <yoctozepto> (exactly what k-a misses ;-) )
15:56:58 <yoctozepto> and how
15:56:59 <mgoddard> it's covered as well as in k-a :)
15:57:06 <yoctozepto> ack ;-)
15:57:13 <mgoddard> let's move on
15:57:16 <mgoddard> #topic XenAPI deprecated in Nova Train, removal pending
15:57:16 <yoctozepto> 3 minutes
15:57:19 <yoctozepto> indeed let's
15:57:21 <yoctozepto> quickie from me
15:57:29 <yoctozepto> we seem to use xenapi with our xen integration
15:57:33 <yoctozepto> I want you to confirm that
15:57:35 <yoctozepto> (or deny)
15:57:44 <mgoddard> looks like it
15:57:44 <yoctozepto> and then we deprecate and remove following nova
15:57:46 <yoctozepto> thank you
15:57:56 <openstackgerrit> Michal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/708114
15:58:03 <mgoddard> certainly lots of greps for xenapi in k-a
15:58:05 <yoctozepto> well, that was fast
15:58:11 <yoctozepto> mnasiadka: your thoughts?
15:58:28 <yoctozepto> jovial[m], priteau, dougsz, osmanlicilegi: any kolla xen experience?
15:58:31 <mgoddard> are they removing completely, or extracting into a separate lib?
15:58:39 <dougsz> nope
15:58:44 <yoctozepto> mgoddard: they deprecated for removal in future
15:58:45 <mnasiadka> it was not really clear what are they doing
15:58:46 <osmanlicilegi> I'm not using xen since 2015 :]
15:58:47 <priteau> none, sorry
15:58:48 <jovial[m]> afraid not, I'ma  libvirt only man
15:58:50 <yoctozepto> in *Train*
15:58:54 <mnasiadka> we don't have xen CI
15:59:00 <mnasiadka> there was somebody that was trying to make it work
15:59:02 <egonzalez> cberent (non online now) i believe he o some of his clients were using xen
15:59:08 <yoctozepto> mnasiadka: thanks, sherlock!
15:59:13 <mgoddard> yeah, JamesBenson maybe?
15:59:24 <mnasiadka> mgoddard: that was the guy - not overly active now
15:59:25 <yoctozepto> wonder how they doing
15:59:26 <mgoddard> hi egonzalez :)
15:59:41 <yoctozepto> egonzalez: what is full name of cberent?
15:59:53 <mgoddard> maybe we should wait for nova removal?
15:59:55 <yoctozepto> (or was)
15:59:56 <dougsz> Is there time for a quick question about /etc/security/limits.conf?
16:00:02 <mnasiadka> mgoddard: I would just deprecate for now
16:00:10 <egonzalez> yoctozepto: Christian Berendt
16:00:12 <mnasiadka> with some nice warning message when enabled
16:00:17 <mgoddard> yoctozepto: berendt is his IRC nick
16:00:23 <mgoddard> deprecation makes sense
16:00:29 <yoctozepto> egonzalez, mgoddard: oh, that's the guy who started helping with reviews again
16:00:34 <yoctozepto> mnasiadka: ^
16:00:36 <mgoddard> (which is what yoctozepto said)
16:00:40 <mgoddard> yes
16:00:44 <mnasiadka> yeah, Christian got a bit active today
16:01:07 <mgoddard> #topic open discussion
16:01:11 <mgoddard> dougsz has a qq
16:01:19 <yoctozepto> nice reviews, we should keep him in core team as we can't add him more
16:01:49 <dougsz> 1. deployment uses base image with some settings in /etc/security/limits.conf
16:01:52 <generalfuzz> If we could have a couple eyes on https://review.opendev.org/664516 it would be greatly appreciated
16:01:52 <patchbot> patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 19 patch sets
16:01:53 <yoctozepto> as for xen - I'll post it to ml
16:02:01 <dougsz> 2. Settings make their way into nova-ssh container
16:02:16 <mnasiadka> (they make their way to all container I guess)
16:02:21 <dougsz> 3. One of the settings is nproc, the number of processes a user can open
16:02:31 <dougsz> 4. That setting restricts the number of SSH sessions
16:02:55 <dougsz> So.. should the Nova SSH container own that file, to prevent this?
16:03:05 <yoctozepto> generalfuzz: we can't handle that root thingy with current sudo?
16:03:12 <dougsz> Should we support customising it explicity?
16:03:33 <dougsz> Or just use custom config in the Docker file to modify it in the Nova SSH container?
16:03:38 <yoctozepto> generalfuzz: I mean, it breaks current assumptions we use sudo to gain root in containers
16:03:40 <mnasiadka> dougsz: What is their setting of max concurrent live migrations?
16:03:42 <cosmicsound> dougsz , +1 that could be also a security issue
16:03:49 <mgoddard> dougsz: I think the problem is that VMs and other processes are created as the nova user, which is also used by the nova-ssh container
16:04:20 <dougsz> yeah
16:04:28 <mnasiadka> mgoddard: I don't know if VMs and other processes use a workflow to get pam_limits in the way, or not
16:04:35 <mnasiadka> mgoddard: without using dimensions it's a tricky thing
16:04:39 <yoctozepto> rebuild is pain for this, let's make it customizable too
16:05:35 <mnasiadka> well, another thing - why not use docker dimensions a.k.a. docker run --ulimit for that?
16:06:06 <generalfuzz> yoctozepto: I will investigate it with sudo
16:06:07 <mgoddard> mnasiadka: I imagine that would be a different cgroup-based limit
16:06:28 <yoctozepto> generalfuzz: thanks, that would limit the scope of the patch, otherwise we need splitting AGAIN :D
16:06:49 <mnasiadka> mgoddard: are you sure? https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit
16:07:09 <mnasiadka> well, dimensions are other thing - I confused everybody I think :)
16:07:54 <mnasiadka> mgoddard: this way instead of copying file, we could set Ulimits via Docker API
16:08:05 <mgoddard> mnasiadka: dimensions == ulimit
16:08:13 <mgoddard> that would be nicer
16:08:25 <mgoddard> and also easy to verify
16:08:36 <openstackgerrit> Will Szumski proposed openstack/kolla-ansible master: Introduce influxdb_datadir_volume  https://review.opendev.org/707861
16:08:51 <mnasiadka> mgoddard: well, I thought dimensions == --cpu-percent and others from Docker CLI - then I confused myself even more :)
16:09:03 <yoctozepto> oh my
16:09:10 <yoctozepto> so much confusion
16:09:11 <yoctozepto> :S
16:09:32 <mgoddard> https://docs.openstack.org/kolla-ansible/rocky/reference/resource-constraints.html
16:09:36 <mnasiadka> I would ask why it's called dimensions instead of ulimits :)
16:09:49 <mnasiadka> mgoddard: so then I'm right
16:09:52 <mgoddard> doesnhttps://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html
16:09:57 <mgoddard> https://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html
16:09:58 <mnasiadka> these are not ulimits, just cgroups resource constraints
16:10:02 <mnasiadka> so I was thinking correct
16:10:03 <mgoddard> has ulimits
16:10:10 <mgoddard> added in stein?
16:10:26 <mnasiadka> then it's an easy thing
16:10:36 <mnasiadka> just use ulimits in the dimensions
16:10:45 <mgoddard> yes. question is whether it should be default or local config
16:10:48 <mnasiadka> instead of doing it with pam_limits
16:11:28 <mnasiadka> mgoddard: well then, it's up to the user/operator - they can use a default/global and only override locally on nova_ssh (or nova_*)
16:12:09 <mgoddard> yeah, although it was quite a nasty bug to track down - maybe we should fix it for everyone? or provide a tuneable?
16:12:52 <mnasiadka> care for an example? The only thing I can think of is warning if there are constrains defined in pam_limit :)
16:13:47 <yoctozepto> so outside docker ulimit overrides limits from internal security?
16:13:58 <yoctozepto> (I lost track of discussion due to all confusion)
16:14:01 <mnasiadka> yoctozepto: well now it's a good question, which one wins
16:14:04 <mnasiadka> :)
16:14:10 <yoctozepto> ah-ha
16:14:44 <mnasiadka> Be careful setting nproc with the ulimit flag as nproc is designed by Linux to set the maximum number of processes available to a user, not to a container. For example, start four containers with daemon user.
16:14:50 <mnasiadka> this is also a good note from the documentation
16:14:56 <mnasiadka> maybe we should add it to our docs as well
16:15:15 <mnasiadka> and recommend using Docker API based ulimits, instead of delivering that file
16:15:25 <mnasiadka> that in a file
16:15:35 <mgoddard> this is why VM procs in nova_libvirt affect nova_ssh
16:16:01 <mgoddard> I think some investigation needs to be done into how to fix this
16:16:08 <yoctozepto> use a separate user?
16:16:18 <mnasiadka> how? :)
16:16:18 <mgoddard> same user - nova
16:16:35 <mgoddard> although I would exect UID namespaces to differ
16:16:50 <mgoddard> seems PAM doesn't care :)
16:16:55 <yoctozepto> mnasiadka: I'm adding confusion, don't confuse me
16:17:08 <yoctozepto> hmm, odd
16:17:11 <mnasiadka> mgoddard: well, it's PAM...
16:17:18 <mgoddard> I think there are too many unknowns to discuss this now
16:17:21 <yoctozepto> needs investigation and let's end the meeting
16:17:25 <mgoddard> someone needs to investigate and report back
16:17:37 <mgoddard> are you picking that up dougsz?
16:17:38 <yoctozepto> dougsz?
16:17:51 <mnasiadka> yes, dougsz came with a question, so he should investigate :D
16:17:57 <dougsz> I will start by writing a bug report :)
16:18:02 <yoctozepto> mnasiadka: I wrote that at the same time ;-)
16:18:11 <yoctozepto> mnasiadka: it's asking dougsz ;-)
16:18:14 <dougsz> and do some investigation
16:18:23 <yoctozepto> dougsz: bug report +1
16:18:26 <mgoddard> #action dougsz to write bug report about nova SSH nproc issue
16:18:27 <yoctozepto> otherwise no problem
16:18:28 <yoctozepto> :-)
16:18:33 <dougsz> thanks guys
16:18:44 <yoctozepto> thanks dougsz
16:19:07 <mgoddard> 18 minutes over, what chatterboxes
16:19:10 <mgoddard> thanks all
16:19:12 <mgoddard> #endmeeting