14:04:01 <irenab> #startmeeting kuryr
14:04:02 <openstack> Meeting started Mon Nov  6 14:04:01 2017 UTC and is due to finish in 60 minutes.  The chair is irenab. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:04:03 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:04:05 <openstack> The meeting name has been set to 'kuryr'
14:04:14 <ltomasbo> o/
14:04:22 <janonymous> o/
14:04:36 <irenab> hi guys
14:04:54 <irenab> anyone else for the weekly?
14:04:59 <dulek> o/
14:05:52 <leyal> o/
14:06:07 <irenab> I guess dmellado and apuimedo won't join since they are at OS summit
14:06:25 <irenab> #topic kuryr
14:07:06 <irenab> Anyone have updates related to general kuryr?
14:07:43 <irenab> moving on then
14:07:54 <irenab> #topic kuryr-libnetwork
14:08:22 <irenab> anything to discuss related to kuryr-libnetwork?
14:08:56 <irenab> moving on
14:09:06 <irenab> #topic kuryr-kubernetes
14:09:43 <dulek> Shall I start with CNI daemon status?
14:09:50 <irenab> dulek, yes, please
14:10:14 <dulek> https://review.openstack.org/#/c/515186/ - initial patch has +2 from apuimedo and I think irenab is testing it.
14:10:37 <dulek> I'm working on support for running kuryr-daemon containerized. I'll be pushing a patch soon.
14:10:40 <irenab> dulek, correct, will finalize it today
14:10:46 <dulek> And I'll need to update and rebase the documentation patch.
14:11:33 <dulek> And that will be it! I'll need to do a few lower priority follow up patches that fix up corner cases and bugs that are now being visible when using CNI daemon.
14:11:42 <irenab> dulek, any plans about the gate for cni split?
14:12:08 <dulek> irenab: That's a very good question. How about I'll try to fix our tempest gates first?
14:12:16 <dulek> Currently does are constantly failing.
14:12:26 <irenab> dulek, :-), totally agree on priorities
14:12:41 <dulek> Once we have gates functional it'll be easier to add it. :)
14:13:01 <irenab> but we need to have gate to make sure its stable to switch cni split to default in devstack
14:13:20 <dulek> irenab: I totally agree, thank you for reminding that.
14:13:39 <irenab> and then probably deprecate the original support
14:14:01 <irenab> dulek, thank you for the update
14:14:14 <irenab> anyone who can test the patch, please do so
14:14:24 <irenab> #link https://review.openstack.org/#/c/515186/
14:14:38 <ltomasbo> sure, I'm happy to test the folow up one (mix containerized and split)
14:14:48 <ltomasbo> I've already tested cni split and works fine!
14:14:56 <irenab> ltomasbo, perfect
14:15:21 <irenab> ltomasbo, any update on stuff you are working on?
14:15:29 <ltomasbo> this is ready for reviews: https://review.openstack.org/#/c/510157/
14:16:01 <ltomasbo> and I'm working on an OOM problem at ODL when using it with kuryr
14:16:17 <irenab> OOM?
14:16:23 <ltomasbo> out of memory
14:16:36 <irenab> on ODL or kuryr side?
14:16:49 <ltomasbo> in ODL mostly
14:17:03 <ltomasbo> but increase the chances due to being deployed with devstack
14:17:18 <ltomasbo> as the java memory is limited to 512MB (instead of 2GB)
14:17:35 <ltomasbo> should be fine from the kuryr side
14:18:04 <ltomasbo> I'm digging also in some problems (probably on kuryr side, but most probably on docker/kubernetes)
14:18:13 <ltomasbo> regarding containers taking long time to boot up
14:18:32 <irenab> #action irenab apuimedo review the patch https://review.openstack.org/#/c/510157/
14:18:39 <ltomasbo> long time until the first one is up, in the nested case
14:18:52 <irenab> ltomasbo, including the image load or when image is local?
14:18:59 <ltomasbo> but still didn't find the culprit (though I know I'm affected by a couple of bugs)
14:19:10 <ltomasbo> irenab, once the image is already there
14:19:18 <ltomasbo> as well as the ports are present in the pool
14:19:35 <ltomasbo> so, it should be faster
14:19:49 <ltomasbo> and it takes more than a minute for the first ocntainer when booting 100 at once
14:19:58 <irenab> interesting , any idea where time is spent?
14:20:22 <ltomasbo> digging a bit it seems I hit this: https://bugzilla.redhat.com/show_bug.cgi?id=1425278
14:20:23 <openstack> bugzilla.redhat.com bug 1425278 in docker ""SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)" error message in logs" [Urgent,New] - Assigned to dwalsh
14:20:24 <irenab> I wonder if the CNI split may improve or it is on the controller side
14:20:35 <ltomasbo> and this https://bugzilla.redhat.com/show_bug.cgi?id=1267291
14:20:35 <openstack> bugzilla.redhat.com bug 1267291 in openvswitch "[Openvswitch] balance-tcp bond mode causing issues with OSP Deployments" [High,Closed: currentrelease] - Assigned to nyechiel
14:21:01 <ltomasbo> and I disabled the os_vif.plug to test if that was also adding some time, but it was not
14:21:53 <dulek> ltomasbo: Ah, commenting out os_vif.plug created an issue for me in OVS on baremetal case.
14:22:08 <irenab> ltomasbo, please report the issue as kuryr bug
14:22:22 <ltomasbo> dulek, not an issue for the nested case
14:22:30 <ltomasbo> as the plug basically does a 'pass'
14:22:40 <ltomasbo> it removed it just to about the privsep thing
14:22:47 <ltomasbo> but it is not helping, so I set it back
14:22:51 <dulek> ltomasbo: Okay, I would need to dig more to understand that. :P
14:23:14 <ltomasbo> irenab, I'm not sure it is a kuryr bug, I need to dig a bit more to figure out what to report...
14:23:32 <ltomasbo> irenab, and the OOM came on my way while debugging...
14:23:55 <ltomasbo> as soon as I understand a bit more about the issue, I'll open a bug!
14:24:00 <irenab> ltomasbo, I wonder if this happens only for the bulk or on the single or lets say 2 Pods  spawing
14:24:22 <ltomasbo> irenab, it is somehow proportional to the amount of pods being created
14:24:32 <ltomasbo> I have 3 worker VMs
14:24:49 <ltomasbo> and if I create 3 containres (on on each VM) it takes around 5-8 seconds to start the first one
14:25:03 <ltomasbo> if I create 30, it takes around 20-30 seconds to start the first one
14:25:13 <ltomasbo> and if it is 100, it takes around 70 seconds
14:25:27 <ltomasbo> so, my bet is on something we do for each container
14:25:42 <irenab> and you sure its on kuryr side?
14:25:54 <ltomasbo> but not sure if it is at the controller (gettting the subnet information) or at the cni side
14:26:06 <ltomasbo> irenab, I'm not sure about that
14:26:13 <ltomasbo> it may not even be on kuryr side
14:26:41 <ltomasbo> I'll dig more during this week and let you know if I find it
14:26:44 <irenab> I wonder if there is some scale impact in case of native k8s
14:26:50 <irenab> ltomasbo, thanks!
14:27:18 <ltomasbo> irenab, it could be on k8s, yes
14:27:38 <ltomasbo> but we haven't seen that on the scale testing we did a couple of months ago
14:27:52 <irenab> #action ltomasbo to investigate case with large number of containers and update on findings
14:27:54 <ltomasbo> perhaps it is related to the OOM that I was hitting
14:28:24 <irenab> scale was with ovs and you see the issue with ODL?
14:28:29 <ltomasbo> so, it may well be ODL
14:28:30 <ltomasbo> ODL
14:28:42 <ltomasbo> scale test was done with OVN
14:28:51 <irenab> I will try to see if can run similar test with Dragonflow
14:28:52 <ltomasbo> and I'm doing it with ODL
14:29:04 <ltomasbo> irenab, it would be great to test that
14:29:32 <irenab> #action irenab try to run scale test for kuryr+dragonflow, nested
14:29:44 <ltomasbo> I can help you recreating my env if you need help (it was a devstack base multinode deployment
14:29:58 <ltomasbo> with 4 VMs (1 master + 3 workers)
14:30:22 <irenab> ltomasbo, would appreciate your help. I guess you have some heat stack for that, right?
14:30:41 <ltomasbo> yep, I'm using a kuryr_heat_pike to create the VMs
14:31:07 <irenab> ltomasbo, I will sync with you offline to get the details
14:31:10 <ltomasbo> and then an ansible-based script to install openshift on top of the VMs
14:31:12 <ltomasbo> sure!
14:31:13 <ltomasbo> thanks!
14:31:28 <irenab> ltomasbo, thank you for the update
14:31:40 <ltomasbo> that's all from my side
14:31:49 <irenab> leyal, would you like to update regarding network policy progress?
14:32:02 <leyal> yes
14:32:26 <irenab> please go ahead
14:32:38 <leyal> I created (with a lot help from irenab) a draft for detailed-design for supporting network-poilcy , will be happy for reviews on that ..
14:32:55 <leyal> https://docs.google.com/document/d/1GShzI4DemoraZdjnpZe9ug1GI9xgl3JcIyjnllTtQN4/edit?usp=sharing
14:33:07 <irenab> #link https://docs.google.com/document/d/1GShzI4DemoraZdjnpZe9ug1GI9xgl3JcIyjnllTtQN4/edit?usp=sharing
14:33:39 <irenab> leyal, any specific issues/questions you would like to discuss now?
14:33:58 <leyal> Hope to upload patch with spec soon.
14:34:49 <leyal> Lets discuses in the draft/spec(when it's will be ready) ..
14:35:02 <ltomasbo> great! I'll read it and try to provide some feedback
14:35:28 <irenab> gdoc has very detailed information regarding the Network Policy support, so anyone who has some spare cycles please take a look before leyal uploads the rst
14:35:51 <irenab> ltomasbo, thanks!
14:35:53 <leyal> ltomasbo , thanks
14:36:24 <irenab> anyone else on kuryr-kubernetes topics?
14:36:59 <yboaron> I can update about my progress with openshift route
14:37:14 <irenab> yboaron, go ahead
14:38:12 <yboaron> started to work on integrating openshift route support with KURYR-K8S , I will share a design doc for review in the next few days
14:38:42 <irenab> yboaron, openshift route is like Ingress Controller or something else?
14:38:57 <yboaron> irenab, right
14:39:35 <irenab> yboaron, is there any launchpad bp for this?
14:40:30 <yboaron> I'll open one , in a very high level KURYR should translate route objects into lbaas L7-policy/pool resources
14:41:10 <irenab> yboaron, great, looking forward to see the details
14:41:29 <yboaron> that's it , I will open a bp , and will share a design doc soon
14:41:57 <irenab> I plan to fix the https://bugs.launchpad.net/kuryr-kubernetes/+bug/1723938
14:41:58 <openstack> Launchpad bug 1723938 in kuryr-kubernetes "Cannot access service of LoadBalancer type " [High,New] - Assigned to Irena Berezovsky (irenab)
14:42:25 <irenab> hope to get it fixed by next week
14:42:25 <ltomasbo> irenab, is that just a security group configuration?
14:43:21 <irenab> yes, but seems to be done upon service creation but not in advance as with other sec. groups configuration
14:44:19 <irenab> the fix should be quite trivial. And the funny thing it works without the fix with reference neutron implementation
14:44:27 <ltomasbo> ohh, true
14:44:38 <ltomasbo> now I remember
14:44:59 <ltomasbo> did you find out why it works with default ml2/ovs?
14:45:03 <ltomasbo> it is a bug?
14:45:11 <yboaron> irenab, same solution for ha-proxy and octavia ?
14:45:52 <irenab> octavia sets proper SGs, so the additional SG configuration will be required only for HA Proxy
14:46:37 <irenab> ltomasbo, I think I checked, but do not remember ...
14:46:49 <ltomasbo> xD
14:47:00 <ltomasbo> same here... maybe even you already mentioned on kuryr channel...
14:47:01 <irenab> the issue is only when FIP is assigned for a vIP
14:47:23 <irenab> ltomasbo, I will check, maybe the details are saved :-)
14:47:56 <irenab> anything else for k8s support?
14:48:38 <irenab> #topic open discussion
14:49:29 <irenab> Well, looks like all of us are pretty occupied with k8s support :-)
14:50:16 <ltomasbo> xD
14:50:23 <irenab> if no one has topic to discuss, I think we can close a meeting
14:50:50 <irenab> thanks everyone for joining
14:51:02 <irenab> #endmeeting