15:00:16 #startmeeting kolla 15:00:16 Meeting started Wed Feb 19 15:00:16 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:17 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:19 The meeting name has been set to 'kolla' 15:00:19 #topic rollcall 15:00:21 \o 15:00:21 o/ 15:00:37 o/ 15:00:52 \o 15:01:03 o/ 15:01:06 o/ 15:02:17 #topic announcements 15:02:31 #undo 15:02:32 Removing item from minutes: #topic announcements 15:02:34 #topic agenda 15:02:40 * Roll-call 15:02:42 * Announcements 15:02:44 * Review action items from last meeting 15:02:46 * CI status 15:02:48 * Ussuri release planning (kolla & kolla ansible) 15:02:50 * Ussuri release planning (kayobe) 15:02:52 ** XenAPI deprecated in Nova Train, removal pending 15:02:54 * Kolla SIG (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig 15:02:56 #topic announcements 15:03:03 Looks like we have none on the agenda. Anyone have any? 15:03:11 nope 15:04:54 I was going to raise an issue, but I think I have a workaround. 15:05:06 so, nope, for now at least 15:05:13 We can discuss in open discussion if you like 15:05:20 #topic review action items from last meeting 15:05:32 yoctozepto to deprecate mongodb for ubuntu, drop for centos 15:05:46 I think I saw it proposed. 15:06:01 good work 15:06:04 #topic CI status 15:06:31 "general issues with new py, affects tripleo as well, and jobs used from k-a" 15:06:40 what is this? 15:06:43 still an issue? 15:08:19 anyone? 15:08:36 I'm not sure about the details 15:08:36 kolla CI looks green 15:09:12 date is 13/2/2020, possibly an old one 15:09:20 we got past that 15:09:25 cool 15:09:30 yeah, but still the workaround is applied I guess 15:09:36 with installing newer virtualenv or something 15:09:37 it is 15:09:43 testing revert now 15:09:45 oh it's that thing 15:09:58 looks passy 15:10:01 https://review.opendev.org/#/c/707623/ 15:10:01 patch 707623 - kolla - Revert "Upgrade virtualenv in pre" - 1 patch set 15:10:19 yoctozepto: kolla-ansible part as well? :) 15:10:35 well, you won't see from here unless you have zuul plugin in browser 15:10:36 yeah 15:10:38 ok, let's try to get those merged 15:10:47 onto k-a 15:10:52 stein ubuntu binary fails on neutron migrations (foreign key blah blah) 15:11:03 it failed recently yeah 15:11:11 I remember seeing it 15:11:17 is it a persistent failure? 15:11:28 let's check :-) 15:11:53 https://zuul.opendev.org/t/openstack/builds?job_name=kolla-ansible-ubuntu-binary&branch=stable%2Fstein 15:12:17 looks bad 15:12:25 anyone want to investigate? 15:12:25 neutron backported the fail to stein? :D 15:12:52 rechecking https://review.opendev.org/#/c/706078/ 15:12:53 patch 706078 - kolla-ansible (stable/stein) - Fix multiple issues with MariaDB handling - 1 patch set 15:12:57 mnasiadka: no idea 15:13:04 mnasiadka: somehow only binary 15:13:07 I wonder why it's binary only. Maybe the package needs a bump? 15:13:13 I already raised two bugs to neutron, not gonna look into third one :) 15:13:15 or stein needs a release 15:13:23 in uca 15:13:26 yes 15:13:38 but they surely should know theu have b0rken debs 15:13:58 let's se 15:14:01 we could check the neutron version in UCA stein-testing - if there's a version difference 15:14:31 sounds like a volunteer 15:15:03 #action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary 15:15:07 thanks :) 15:15:28 report back if it's more complicated than that 15:15:42 Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 15:15:59 "general issues with new py when run from kolla side" 15:16:03 Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 15:16:08 assume this is the virtualenv issue? 15:16:27 mgoddard: the same 15:16:33 removing 15:16:39 mgoddard: 14.04 got to updates some time ago, probably it's broken-ish 15:16:51 14.04?? 15:16:59 neutron-server 14.04 15:17:05 whatever it is on Ubuntu side :) 15:17:10 lol, thought you meant ubuntu 14.04 15:17:12 http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html 15:17:36 mnasiadka: and what is current 15:18:06 14.1.0 15:18:18 I don't like the 1 in the middle :) 15:18:24 14.0.4 vs 14.1.0 15:18:27 mnasiadka: why? 15:18:44 it's a bumpy in reqs most likely 15:18:46 it basically means normally it's a big change or some feature backport :) 15:19:05 or exactly fix for our issue which required dep update ;-) 15:19:19 anyways, mnasiadka pokes uca and we are good 15:19:20 might be 15:19:23 #undo 15:19:24 Removing item from minutes: #link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html 15:19:28 well, another bug to raise I guess 15:19:30 #undo 15:19:32 Removing item from minutes: #action mnasiadka to check ubuntu stein UCA neutron versions and request release if necessary 15:19:36 #link http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/stein_versions.html 15:19:52 #action mnasiadka request neutron 14.1.0 in stein UCA 15:20:01 pretty precise 15:20:13 precise pangolin? 15:20:24 anyways 15:20:39 xD 15:20:41 #topic Ussuri release planning (kolla & kolla ansible) 15:21:06 Any Ussuri features we need to discuss this week? 15:21:52 maybe not a feature, but a community goal - can I get some reviews on https://review.opendev.org/#/c/707800/ ? 15:21:52 patch 707800 - kolla - [community goal]: Add contributor and PTL guide - 2 patch sets 15:22:24 will try to get a looko 15:22:31 good 15:22:33 "So You Want to Contribute..." is in the goal? ;D 15:22:45 would love some reviews on OVN patch 15:22:57 my review queue seems to be growing faster than I can process 15:22:59 yoctozepto: complain to the goal coordinator :) 15:23:09 mnasiadka: I started a review of OVN :) 15:23:31 mgoddard: good to hear :) 15:23:35 ah I feel ashame now. mnasiadka I'm so sorry I couldn't spent time for the comments you wait from me :| 15:23:40 will try to finish 15:24:22 osmanlicilegi: easy, fix your datacenter :) 15:24:27 can we mark https://blueprints.launchpad.net/kolla-ansible/+spec/ceph-ansible complete? 15:24:51 well, I'd like to add prechecks for keys existence - but it's rather optional and nice to have 15:24:58 and https://blueprints.launchpad.net/kolla-ansible/+spec/remove-ceph 15:25:10 yeah, that one for sure 15:25:15 mnasiadka: the blueprint is just about CI 15:25:36 mgoddard: right, I just took the extra mile to make it more user friendly :) 15:25:57 always appreciated 15:25:59 whoa, remove-ceph was never approved :) 15:26:20 mnasiadka: WHAT HAVE YOU DONE?! 15:26:55 yoctozepto: well, it's better to beg for forgiveness, than to ask for permission - that's my life motto :) 15:27:09 mnasiadka: gets things done faster 15:27:12 rm -rf; echo sorry 15:27:27 yeah, sorry is the key 15:27:28 mgoddard: I've almost spilled my drink 15:27:48 apologies 15:28:12 although I did already say sorry 15:29:19 anything else to discuss? 15:29:39 only stuff for later 15:29:52 #topic Ussuri release planning (kayobe) 15:30:02 dougsz 15:30:10 Can't see priteau or jovial 15:30:56 where them folks at 15:30:59 where, where 15:31:19 hmm.. what do we have, possibly a cells config cleanup 15:32:04 dougsz: what needs to be done for that? 15:32:18 So ceph stays? :D 15:32:29 * cosmicsound chuckels 15:32:41 cosmicsound: ceph? too late - it's gone 15:32:43 Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 15:32:51 aha, priteau & jovial[m] 15:32:58 o/ 15:33:07 hi 15:33:12 mgoddard: Anything to make it more manageable for high numbers of cells 15:33:50 on the topic of ceph.. do we have any Kayobe references to remove? 15:34:07 there is more work cells planned for k-a, will you be looking at it before ussuri? 15:34:38 We need shared cell controllers sooner or later 15:34:45 I will try to 15:34:52 Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add linuxbridge jobs https://review.opendev.org/708318 15:35:00 We already have some half backed patches around that 15:35:07 yep 15:35:15 *baked 15:35:27 I think a separate nova-compute role might make it more scalable, but that needs investigation 15:35:41 agreed 15:35:49 on the ceph removal, I created https://storyboard.openstack.org/#!/story/2007295 15:36:02 a high priority for me would be to implement the custom extension points proposal 15:36:03 we have a role to label block devices 15:36:17 anyone want to pick up ceph removal? 15:36:31 should be therapeutic if you like removing code 15:36:52 I will volunteer if no one else wants it 15:37:22 #action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 15:37:23 thanks 15:37:25 thanks jovial[m] 15:37:26 mgoddard: is it possible to volunteer from outside? 15:37:30 ah, too late :-) 15:37:41 anyone can volunteer to work on kayobe 15:37:50 there is no outside in open source :) 15:37:55 well, you know, making it official 15:37:57 and stuff 15:38:16 could get to know kayobe better 15:38:38 you want to take it? 15:38:47 I'm sure jovial[m] won't mind too much :) 15:38:53 mgoddard: yeah, therapeutic 15:38:59 #undo 15:39:00 Removing item from minutes: #action jovial[m] to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 15:39:09 fine by me :) 15:39:10 #action yoctozepto to remove kayobe ceph block device labelling support https://storyboard.openstack.org/#!/story/2007295 15:39:13 thanks yoctozepto 15:39:19 +1 15:39:19 ;-) 15:39:37 jovial[m] is away for a while anyway 15:40:09 jovial[m]: you would like to see the custom extension points proposal 15:40:30 https://storyboard.openstack.org/#!/story/2001663 15:40:30 :) 15:40:32 We certainly have an issue with exporters, would be nice to use it for that 15:40:46 jovial[m]: want to pick it up? 15:41:15 for sure - lots of people want this I believe 15:41:25 +1 15:41:47 it will make for a smoother ride in many cases 15:42:00 Extension points and multiple environments are two big requested features 15:42:02 #action jovial[m] to work on custom extension points 15:42:05 yep 15:42:14 folks, remember some extensions might better fit in k-a 15:42:18 and these are welcome as well 15:42:22 mgoddard to correct me :-) 15:42:33 yoctozepto: you are right 15:42:46 we will try to use judgement 15:42:55 OK 15:43:15 this is about a framework for running hook playbooks at various points in kayobe workflows 15:43:49 mgoddard: yeah, skimmed over story 15:44:01 onto the other big feature then - multiple environments 15:44:13 https://storyboard.openstack.org/#!/story/2002009 15:44:21 anyone interested in picking this up? 15:44:30 it's a bigger task 15:44:39 not enough cycles, sorry 15:44:42 could potentially be a collaboration between multiple people 15:45:32 Maybe someone is willing to help fund it? 15:45:33 I'd like to be involved to some extent, but CentOS 8 is eating my cycles at the moment 15:46:03 another must have feature in my opinion 15:46:15 I think we have a funder (getting a bit internal here though :)) 15:46:31 ok, no takers yet 15:46:35 keep it in mind 15:46:38 mgoddard: slightly off topic; bored with bifrost, how is bikolla going? I really liked the "containerized bifrost" 15:46:40 how long have we got until the next release? 15:47:08 Kolla feature freeze: Apr 27 - May 01 15:47:08 mgoddard: has it any roadmap or whatever? 15:47:10 Kolla RC1: May 11 - May 15 15:47:12 Kolla final release: Jun 1 - Jun 5 15:47:16 can we fit it any place? 15:47:38 when you say containerised bifrost you don't mean our current bifrost container :) 15:47:50 mgoddard: I mean bikolla hence "" 15:47:53 :D 15:48:08 dougsz was looking at it recently 15:48:19 dougsz: how's bikolla? :-) 15:48:29 still at the proposal/design stage though IIRC 15:48:47 or possibly just blocked on cycles :) 15:48:52 I'd like to be in for any eventual discussion 15:48:53 we need it for deploying a large number of hypervisors 15:49:20 I was also wondering about just using the top level ironic for that 15:49:28 noted yoctozepto 15:49:36 thanks 15:49:52 dougsz: what do you mean by "top level" 15:50:04 the overcloud ironic, not bifrost ironic 15:50:08 that is the other option 15:50:32 dougsz: don't you want to use it for standing up the overcloud? :) 15:50:45 yeah, I'm confused too 15:51:00 well, bifrost can still handle a small number of controllers, just not the hypervisors 15:51:02 I thought bikolla is about deploying standalone ironic with kolla 15:51:12 so bifrost via kolla without bifrost 15:51:28 correct 15:51:31 dougsz: looking from Kolla side - I would be happy to deprecate and remove bifrost container, and this way we would need to maintain it :) 15:52:12 +1 15:52:20 yeah, it is balancing the maintenance of bifrost against the work required for bikolla 15:52:21 would kayobe need a name change? ;-) 15:52:45 jovial[m]: not it's a snake 15:52:51 be become bikolla 15:53:04 kayobikolla? 15:53:05 kolla on bi...kolla on bi..kolla ..... 15:53:16 perpetum kayobile 15:53:23 :D 15:53:24 mgoddard: looking from another perspective - maybe it would make sense to document bikolla in k-a and make it easier to use for users? I don't think anybody is really aware of using bikolla instead of bifrost. 15:53:40 ^ that's also what I meant 15:53:48 mnasiadka better at putting my thoughts in words 15:53:52 thanks mnasiadka 15:54:05 we could do that 15:54:23 "but we won't" 15:54:36 well, anybody with spare cycles can do that 15:54:42 mnasiadka to pick it up? 15:54:54 I am too nooby atm ;p 15:54:56 I can add it to my list, although not with a very high priority :) 15:55:26 high is fine ;-) 15:55:29 hell, we could even do deploy-bikolla with some stock inventory file and stock globals to deploy it on localhost 15:55:39 docs to deploy the containers should be easy enough 15:56:12 using the thing and a full tested procedure would take longer 15:56:41 that said, does kayobe have docs on how ci is configured? 15:56:44 aka what is tested 15:56:52 (exactly what k-a misses ;-) ) 15:56:58 and how 15:56:59 it's covered as well as in k-a :) 15:57:06 ack ;-) 15:57:13 let's move on 15:57:16 #topic XenAPI deprecated in Nova Train, removal pending 15:57:16 3 minutes 15:57:19 indeed let's 15:57:21 quickie from me 15:57:29 we seem to use xenapi with our xen integration 15:57:33 I want you to confirm that 15:57:35 (or deny) 15:57:44 looks like it 15:57:44 and then we deprecate and remove following nova 15:57:46 thank you 15:57:56 Michal Nasiadka proposed openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance https://review.opendev.org/708114 15:58:03 certainly lots of greps for xenapi in k-a 15:58:05 well, that was fast 15:58:11 mnasiadka: your thoughts? 15:58:28 jovial[m], priteau, dougsz, osmanlicilegi: any kolla xen experience? 15:58:31 are they removing completely, or extracting into a separate lib? 15:58:39 nope 15:58:44 mgoddard: they deprecated for removal in future 15:58:45 it was not really clear what are they doing 15:58:46 I'm not using xen since 2015 :] 15:58:47 none, sorry 15:58:48 afraid not, I'ma libvirt only man 15:58:50 in *Train* 15:58:54 we don't have xen CI 15:59:00 there was somebody that was trying to make it work 15:59:02 cberent (non online now) i believe he o some of his clients were using xen 15:59:08 mnasiadka: thanks, sherlock! 15:59:13 yeah, JamesBenson maybe? 15:59:24 mgoddard: that was the guy - not overly active now 15:59:25 wonder how they doing 15:59:26 hi egonzalez :) 15:59:41 egonzalez: what is full name of cberent? 15:59:53 maybe we should wait for nova removal? 15:59:55 (or was) 15:59:56 Is there time for a quick question about /etc/security/limits.conf? 16:00:02 mgoddard: I would just deprecate for now 16:00:10 yoctozepto: Christian Berendt 16:00:12 with some nice warning message when enabled 16:00:17 yoctozepto: berendt is his IRC nick 16:00:23 deprecation makes sense 16:00:29 egonzalez, mgoddard: oh, that's the guy who started helping with reviews again 16:00:34 mnasiadka: ^ 16:00:36 (which is what yoctozepto said) 16:00:40 yes 16:00:44 yeah, Christian got a bit active today 16:01:07 #topic open discussion 16:01:11 dougsz has a qq 16:01:19 nice reviews, we should keep him in core team as we can't add him more 16:01:49 1. deployment uses base image with some settings in /etc/security/limits.conf 16:01:52 If we could have a couple eyes on https://review.opendev.org/664516 it would be greatly appreciated 16:01:52 patch 664516 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 19 patch sets 16:01:53 as for xen - I'll post it to ml 16:02:01 2. Settings make their way into nova-ssh container 16:02:16 (they make their way to all container I guess) 16:02:21 3. One of the settings is nproc, the number of processes a user can open 16:02:31 4. That setting restricts the number of SSH sessions 16:02:55 So.. should the Nova SSH container own that file, to prevent this? 16:03:05 generalfuzz: we can't handle that root thingy with current sudo? 16:03:12 Should we support customising it explicity? 16:03:33 Or just use custom config in the Docker file to modify it in the Nova SSH container? 16:03:38 generalfuzz: I mean, it breaks current assumptions we use sudo to gain root in containers 16:03:40 dougsz: What is their setting of max concurrent live migrations? 16:03:42 dougsz , +1 that could be also a security issue 16:03:49 dougsz: I think the problem is that VMs and other processes are created as the nova user, which is also used by the nova-ssh container 16:04:20 yeah 16:04:28 mgoddard: I don't know if VMs and other processes use a workflow to get pam_limits in the way, or not 16:04:35 mgoddard: without using dimensions it's a tricky thing 16:04:39 rebuild is pain for this, let's make it customizable too 16:05:35 well, another thing - why not use docker dimensions a.k.a. docker run --ulimit for that? 16:06:06 yoctozepto: I will investigate it with sudo 16:06:07 mnasiadka: I imagine that would be a different cgroup-based limit 16:06:28 generalfuzz: thanks, that would limit the scope of the patch, otherwise we need splitting AGAIN :D 16:06:49 mgoddard: are you sure? https://docs.docker.com/engine/reference/commandline/run/#set-ulimits-in-container---ulimit 16:07:09 well, dimensions are other thing - I confused everybody I think :) 16:07:54 mgoddard: this way instead of copying file, we could set Ulimits via Docker API 16:08:05 mnasiadka: dimensions == ulimit 16:08:13 that would be nicer 16:08:25 and also easy to verify 16:08:36 Will Szumski proposed openstack/kolla-ansible master: Introduce influxdb_datadir_volume https://review.opendev.org/707861 16:08:51 mgoddard: well, I thought dimensions == --cpu-percent and others from Docker CLI - then I confused myself even more :) 16:09:03 oh my 16:09:10 so much confusion 16:09:11 :S 16:09:32 https://docs.openstack.org/kolla-ansible/rocky/reference/resource-constraints.html 16:09:36 I would ask why it's called dimensions instead of ulimits :) 16:09:49 mgoddard: so then I'm right 16:09:52 doesnhttps://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html 16:09:57 https://docs.openstack.org/kolla-ansible/latest/reference/deployment-config/resource-constraints.html 16:09:58 these are not ulimits, just cgroups resource constraints 16:10:02 so I was thinking correct 16:10:03 has ulimits 16:10:10 added in stein? 16:10:26 then it's an easy thing 16:10:36 just use ulimits in the dimensions 16:10:45 yes. question is whether it should be default or local config 16:10:48 instead of doing it with pam_limits 16:11:28 mgoddard: well then, it's up to the user/operator - they can use a default/global and only override locally on nova_ssh (or nova_*) 16:12:09 yeah, although it was quite a nasty bug to track down - maybe we should fix it for everyone? or provide a tuneable? 16:12:52 care for an example? The only thing I can think of is warning if there are constrains defined in pam_limit :) 16:13:47 so outside docker ulimit overrides limits from internal security? 16:13:58 (I lost track of discussion due to all confusion) 16:14:01 yoctozepto: well now it's a good question, which one wins 16:14:04 :) 16:14:10 ah-ha 16:14:44 Be careful setting nproc with the ulimit flag as nproc is designed by Linux to set the maximum number of processes available to a user, not to a container. For example, start four containers with daemon user. 16:14:50 this is also a good note from the documentation 16:14:56 maybe we should add it to our docs as well 16:15:15 and recommend using Docker API based ulimits, instead of delivering that file 16:15:25 that in a file 16:15:35 this is why VM procs in nova_libvirt affect nova_ssh 16:16:01 I think some investigation needs to be done into how to fix this 16:16:08 use a separate user? 16:16:18 how? :) 16:16:18 same user - nova 16:16:35 although I would exect UID namespaces to differ 16:16:50 seems PAM doesn't care :) 16:16:55 mnasiadka: I'm adding confusion, don't confuse me 16:17:08 hmm, odd 16:17:11 mgoddard: well, it's PAM... 16:17:18 I think there are too many unknowns to discuss this now 16:17:21 needs investigation and let's end the meeting 16:17:25 someone needs to investigate and report back 16:17:37 are you picking that up dougsz? 16:17:38 dougsz? 16:17:51 yes, dougsz came with a question, so he should investigate :D 16:17:57 I will start by writing a bug report :) 16:18:02 mnasiadka: I wrote that at the same time ;-) 16:18:11 mnasiadka: it's asking dougsz ;-) 16:18:14 and do some investigation 16:18:23 dougsz: bug report +1 16:18:26 #action dougsz to write bug report about nova SSH nproc issue 16:18:27 otherwise no problem 16:18:28 :-) 16:18:33 thanks guys 16:18:44 thanks dougsz 16:19:07 18 minutes over, what chatterboxes 16:19:10 thanks all 16:19:12 #endmeeting