14:00:16 <irenab> #startmeeting kuryr
14:00:16 <openstack> Meeting started Mon Feb 19 14:00:16 2018 UTC and is due to finish in 60 minutes.  The chair is irenab. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:20 <openstack> The meeting name has been set to 'kuryr'
14:00:37 <leyal> Hi
14:00:40 <dulek> Hello!
14:00:54 <maysamacedos> Hi
14:01:27 <irenab> Let's wait few more minutes for others to join
14:01:47 <kaliya> Greetings
14:02:04 <ltomasbo> o/
14:02:44 <irenab> Hi all, thank you for joining. Let's start
14:02:56 <irenab> #topic kuryr-libnetwork
14:02:58 <yboaron> o/
14:03:13 <garyloug> o/
14:04:16 <irenab> there is a patch #link https://review.openstack.org/#/c/544548/ that deals with tags extension changes in neutron
14:05:00 <irenab> the question is if previous extensions, meaning previous neutron versions should be supported
14:05:18 <celebdor> irenab: I'm inclined to say that they should
14:05:27 <celebdor> is it a lot of trouble to make it compatible with both?
14:05:41 <irenab> actually there are 3 options
14:06:02 <dulek> tag and tag_ext are deprecated since pike: https://docs.openstack.org/releasenotes/neutron/pike.html#deprecation-notes
14:06:20 <dulek> And guys planned to remove it in Queens, but left it. It's getting removed in Rocky.
14:06:22 <irenab> It does not look like a big deal
14:06:57 <dulek> So question is do we want Rocky's Kuryr to support Ocata?
14:07:07 <dulek> s/do we want/do we care
14:07:41 <celebdor> dulek: it only depends on the cost
14:07:56 <irenab> I believe the cost is not big
14:08:13 <irenab> #chair celebdor
14:08:14 <openstack> Current chairs: celebdor irenab
14:08:33 <irenab> so shall we vote for keeping both?
14:08:47 <celebdor> if the cost is not big, since we are not using any new features of the new tags
14:08:51 <celebdor> we should support them
14:08:56 <irenab> +1
14:09:02 <irenab> dulek ?
14:09:31 <dulek> Works for me either way. It's not a lot to carry in the code.
14:09:40 <irenab> great
14:09:53 <irenab> any other libnetwork related issues to discuss?
14:10:33 <irenab> ok, moving on
14:10:41 <irenab> #topic kuryr-kubernetes
14:11:03 <irenab> anyone want to update?
14:11:26 <celebdor> man...
14:11:37 <celebdor> Well, I filed two bugs last friday
14:11:46 <irenab> links?
14:11:55 <celebdor> basically with neutron-lbaasv2 and firewall=openvswitch
14:11:58 <celebdor> services are broken
14:13:24 <celebdor> just a sec
14:13:30 <yboaron> #link https://bugs.launchpad.net/kuryr-kubernetes/+bug/1749968
14:13:30 <openstack> Launchpad bug 1749968 in kuryr-kubernetes "services backed by neutron-lbaas do not work with native ovs firewall" [Critical,In progress] - Assigned to Antoni Segura Puimedon (celebdor)
14:13:41 <celebdor> thanks yboaron
14:13:53 <celebdor> the biggest contention point though
14:14:12 <celebdor> is that for east-west, setting the same SG as the pods work
14:14:17 <celebdor> for loadbalancer service type
14:14:42 <celebdor> we either need to dynamically change the ports in a SG per LB depending ont he listeners
14:15:03 <celebdor> or just allow everything in for external traffic when service is loadbalancer type
14:15:10 <irenab> for the issue yboaron raises I was that woith Dragonflow there is a need to add SG for the specific port of the servic itself
14:15:22 <celebdor> since neutron-lbaasv2 is deprected I'm inclined to just allow everything
14:15:39 <celebdor> (specially considering haproxy will only LB the listener ports anyway)
14:15:55 <yboaron> irenab, IIRC it was LoadBalancer service type --> means N-S
14:16:03 <irenab> yes
14:16:52 <irenab> celebdor, the patch you proposed covers what you just explained?
14:17:19 <celebdor> only covers east-west
14:17:26 <celebdor> n-s needs to be added
14:17:30 <celebdor> one of the two options above
14:17:49 <yboaron> #link https://review.openstack.org/#/c/545363/  - should support east-west
14:17:56 <celebdor> (I'd rather not carry the extra SG per LB since it feels like repeating octavia code, but I can be persuaded)
14:18:47 <irenab> the question is if we officially support HAProxy flavor (till it is removed)
14:18:53 <celebdor> irenab: dulek: ltomasbo: yboaron: ^^
14:19:12 <celebdor> irenab: do you have native lb already in df?
14:19:33 <irenab> not yet, speced but not implemented
14:20:11 <dulek> celebdor: I don't have strong opinions here, but with neutron-lbaas being deprecated, I'd say to stick with Octavia's way.
14:20:28 <ltomasbo> neither do I
14:20:49 <ltomasbo> but agree with dulek, give preference to Octavia's way
14:20:57 <irenab> dulek, Octavia is a bit more resource heavy flavor,I won't be surprised if some will choose HAProxy
14:21:23 <celebdor> irenab: so do you use octavia or lbaasv2?
14:21:24 <ltomasbo> irenab, that's true
14:21:39 <irenab> celebdor, use for what?
14:21:48 <irenab> I mean the kuryr users
14:21:49 <celebdor> irenab: in your deployments
14:22:04 <celebdor> I'm inclined to support lbaasv2 until we have either containerized octavia or octavia plugins
14:22:38 <irenab> I agree, then probably need to solve n-s in the Octavia like way
14:22:38 <dulek> celebdor: It's awful that we don't have anything rock-solid in LB space…
14:23:02 <dulek> I mean - we wonder if anyone will use Octavia, while LBaaSv2 is already deprecated.
14:23:09 <dulek> This isn't really healthy situation.
14:23:37 <celebdor> irenab: I was afraid you'd say that
14:23:49 <irenab> dulek, as far as I remember there is no specific details about when lbaasv2 will be removed
14:24:02 <dulek> irenab: It was announced recently.
14:24:04 <celebdor> dulek: Octavia, as is, can only be used at scale for N-S
14:24:14 <celebdor> in the kubernetes scenario
14:24:14 <irenab> celebdor, lets finalize it at PTG?
14:24:24 <celebdor> irenab: well, I'd like to merge it by wednesday
14:24:27 <celebdor> I think it is doable
14:24:30 <celebdor> with a bit of help
14:24:47 <celebdor> irenab: the advantage is that we can take "inspiration" from the octavia code
14:24:48 <dulek> celebdor: That's a release blocker for Queens or not?
14:24:57 <celebdor> dulek: for me it is
14:25:01 <dulek> irenab: http://lists.openstack.org/pipermail/openstack-dev/2018-January/126836.html
14:25:43 <dulek> celebdor: Then we need to have it fixed this week.
14:25:44 <irenab> dulek, "We are not announcing the end of the deprecation cycle at this time"
14:26:04 <irenab> when is the deadline for Q. release?
14:26:06 <dulek> irenab: Uh, oh. Did I simply read what I wanted to read? :P
14:26:08 <celebdor> it's like
14:26:20 <celebdor> we don't want you to use it, but we won't screw you if you do
14:26:25 <celebdor> that's how I read it
14:26:37 <dulek> Q is released next Wednesday. We can have another RC before that date.
14:26:46 <irenab> yes, mainly no new features on lbaasv2, but you can keep using it as is
14:26:55 <celebdor> dulek: this wednesday or next week's
14:26:57 <celebdor> ?
14:27:10 <dulek> celebdor: IIRC next, but let me check again.
14:27:40 <celebdor> thanks dulek
14:27:46 <celebdor> Oh, I feel like I'm PTL again
14:27:48 <celebdor> :D
14:27:54 <dulek> Ah, okay. So we have time until Friday to issue another RC.
14:27:58 <dulek> https://releases.openstack.org/queens/schedule.html#q-finalrc
14:28:10 <irenab> well then, lets try to have a fix and merge it by Friday
14:28:15 <celebdor> let's
14:28:44 <celebdor> do we have anything else?
14:28:49 <celebdor> oh, yes
14:29:00 <celebdor> maysamacedos reported that the cni daemon is leaking memory
14:29:09 <maysamacedos> yup
14:29:11 <celebdor> since she is working on a very cool feature
14:29:21 <celebdor> liveness checks for the cni daemon
14:29:32 <celebdor> that mark unhealthy when we go over certain memory
14:29:41 <maysamacedos> the IPDB more precisely
14:29:48 <celebdor> I fixed in pyroute2 upstream one memory leak
14:29:55 <celebdor> but we know there is another in setns
14:29:58 <celebdor> maybe more
14:30:15 <irenab> interesting
14:30:25 <ltomasbo> umm, nice!
14:30:26 <celebdor> this is only a chance to talk about the liveness memory leak health
14:31:01 <maysamacedos> should we maintain this (IPDB) check?
14:31:12 <dulek> maysamacedos: Had you checked the leak with newer pyroute2 version, including celebdor's fix?
14:31:16 <celebdor> I think we should
14:31:24 <maysamacedos> dulek: yes
14:31:26 <celebdor> dulek: she did
14:31:35 <dulek> And that doesn't solve it? :(
14:31:37 <celebdor> we were discussing it last week
14:31:43 <maysamacedos> dulek: no
14:31:48 <celebdor> dulek: as I said, I know there's at least one more
14:31:56 <celebdor> what I proposed was to have a conf option
14:32:15 <celebdor> that says use at max 8GiB of mem
14:32:23 <celebdor> if you use more it means you are leaking
14:32:26 <celebdor> and you should be killed
14:32:35 <maysamacedos> hmm
14:32:48 <irenab> celebdor, so when this check fails, node cannot host any more Pods?
14:32:49 <celebdor> dulek: which reminds me... Do we always evict entries from the registry?
14:32:50 <dulek> celebdor: Let's just have the limit configurable. -1 means no limit, the you can set it in MiBs.
14:33:04 <celebdor> irenab: no, when it fails the health check fails
14:33:14 <irenab> and this leads to what?
14:33:14 <dulek> irenab: No, kuryr-daemon will get restarted.
14:33:15 <celebdor> and k8s should restart the cni daemon
14:33:21 <celebdor> and then it should start working again :-)
14:33:27 <dulek> Magic. ;)
14:33:28 <irenab> ok
14:33:29 <celebdor> the node will go notready -> ready
14:33:38 <celebdor> maysamacedos: it does restart it, right?
14:33:44 <dulek> celebdor: I don't think that, most likely CNI will not notice.
14:33:47 <maysamacedos> tes
14:33:47 <irenab> so existing Pods are not affected
14:33:53 <maysamacedos> yes celebdor
14:34:21 <celebdor> good
14:34:22 <dulek> celebdor: And about deleting stuff from registry. We currently don't do that, don't even watch for DELETEs.
14:34:26 <celebdor> irenab: they are not
14:34:35 <celebdor> dulek: that's leaking
14:34:42 <celebdor> ok
14:34:44 <celebdor> not really
14:34:45 <dulek> celebdor: I think it's easy to be implemented now, as we have locks now.
14:34:48 <celebdor> it's ballooning
14:34:50 <celebdor> xD
14:34:53 <dulek> :)
14:35:01 <dulek> celebdor: I can try to fix that today.
14:35:03 <celebdor> dulek: please, file a bug
14:35:12 <celebdor> and fix it :P
14:35:23 <irenab> nice troubleshooting session :-)
14:35:42 <irenab> any other issues to discuss?
14:35:42 <celebdor> irenab: I feel like I'm forgetting about some release bug
14:35:49 <celebdor> but we can probably address it by the .1
14:35:53 <celebdor> .0 is always dangerous
14:35:55 <celebdor> :P
14:36:04 <dulek> celebdor: You have two critical issues in launchpad.
14:36:15 <dulek> https://bugs.launchpad.net/kuryr-kubernetes
14:36:38 <celebdor> the lbaas ones
14:36:54 <irenab> actually there are 3
14:37:07 <celebdor> dulek: what about this one https://bugs.launchpad.net/kuryr-kubernetes/+bug/1731485
14:37:07 <openstack> Launchpad bug 1731485 in kuryr-kubernetes "Kuryr ignores CNI_CONTAINERID when serving requests" [Critical,In progress] - Assigned to Michal Dulko (michal-dulko-f)
14:37:11 <celebdor> it has your name on it
14:37:13 <celebdor> :-)
14:37:28 <dulek> celebdor: Yup. I don't know how to fix it for case without kuryr-daemon.
14:37:44 <irenab> https://bugs.launchpad.net/kuryr-kubernetes/+bug/1749921
14:37:45 <openstack> Launchpad bug 1749921 in kuryr-kubernetes "Loadbalancer service type fails to create due to subnet access policy" [Critical,In progress] - Assigned to Yossi Boaron (yossi-boaron-1234)
14:38:00 <dulek> And for kuryr-daemon it's fixed. I think we'll get over it once kuryr-daemon will become a default.
14:38:44 <irenab> https://review.openstack.org/#/c/545270/ is almost ready to be merged
14:39:08 <celebdor> dulek: what do you think is missing to make it default
14:39:11 <irenab> We saw some strange issue with unhealthy lb handler, just need to verify it is resolved
14:39:12 <celebdor> (apart from mem leaks)
14:39:26 <yboaron> irenab, Did you manage to check it ?
14:39:39 <celebdor> irenab: I thought yboaron found out what it was
14:39:44 <celebdor> or you mean with df?
14:39:49 <irenab> yboaron, not the latest version, will complete asap
14:39:55 <dulek> celebdor: Nothing really. We gate on that, we know it solves an issue that we don't know how to approach without kuryr-daemon.
14:40:12 <dulek> celebdor: I'd say it's ready to become a default in Rocky.
14:40:17 <yboaron> It worked for me , crossing fingers :-)
14:40:17 <irenab> celebdor, not with DF, it was some kuryr-kubernetes issue
14:40:26 <celebdor> yboaron: what is that ep_subsets thing?
14:40:33 <celebdor> I don't recall putting that there
14:40:35 <celebdor> what did I miss
14:40:54 <celebdor> dulek: alright
14:41:11 <celebdor> you can put a BP to make it default and deprecate haproxy on Rocky
14:41:12 <yboaron> That was the exception I got , triggers the unhealthy of LB
14:41:24 <celebdor> yboaron: what's it about?
14:41:30 <celebdor> do you have a stactrace?
14:41:42 <irenab> celebdor, I posted it on the patch
14:41:50 <yboaron> iterable of None object
14:42:02 <celebdor> will check, thanks
14:42:15 <dulek> celebdor: HAProxy? I've talked about deploying without kuryr-daemon. :P
14:42:42 <irenab> I think celebdor is ulti tasking as usual, abit of context switch slip ...
14:42:45 <celebdor> dulek: and that's why I meant non daemonized cni
14:42:49 <celebdor> I'm just mistyping
14:42:51 <dulek> :)
14:42:59 <yboaron> I suspect that service doesn't work in latest devstack  - I was in the middle of testing that .. and guess what happened to RDO ?
14:43:00 <celebdor> irenab: only 4 conversations at one
14:43:10 <celebdor> yboaron: Kaboom?
14:43:24 <yboaron> celebdor, Yep
14:43:38 <irenab> yboaron, dmellado mentioned maitenance of RDO
14:44:01 <yboaron> irenab, yes disks upgrade ..
14:44:18 <irenab> Shall we move to open discussions?
14:44:22 <yboaron> Did someone check service LB lately on devstack ?
14:44:31 <celebdor> irenab: agreed
14:44:38 <irenab> yboaron, what do you mean?
14:44:59 <yboaron> clean devstack - create LB E-W
14:45:06 <yboaron> clusterIP
14:45:24 <irenab> not recently
14:45:26 <celebdor> yboaron: on BM?
14:45:27 <yboaron> I saw strange things with the health reports
14:45:42 <celebdor> yboaron: you mean the controller heatlh?
14:45:43 <yboaron> celebdor, VM-devstack
14:45:50 <celebdor> ok
14:45:51 <yboaron> celebdor, yes
14:45:54 <celebdor> that's "baremetal"
14:46:01 <celebdor> for poor people without real hardware
14:46:03 <celebdor> xD
14:46:06 <irenab> yboaron, please open a bug if you see some issues
14:46:21 <yboaron> irenab, I will
14:46:22 <irenab> #topic open discussions
14:46:29 <irenab> #chair dmellado
14:46:29 <openstack> Current chairs: celebdor dmellado irenab
14:46:58 <irenab> dmellado asked me to remind about comming PTG, its going to be next week
14:47:14 <dulek> \o/
14:47:38 <irenab> we have an etherpad where anyone who want to suggest session can add it #link https://etherpad.openstack.org/p/kuryr-ptg-rocky
14:47:41 <ltomasbo> yep, he wanted to plan a bit the schedule https://ethercalc.openstack.org/kuryr_ptg
14:48:26 <irenab> correct, so please if you plan to join remotely add the prefered time slot based on the availability at the ethercalc
14:48:36 <dulek> ltomasbo: Why is 15:30-16:30 not available everywhere?
14:48:38 <celebdor> irenab: theoretically the proposal time is closed
14:48:50 <celebdor> only scheduling now, right?
14:49:03 <celebdor> nap time?
14:49:26 <irenab> :-)
14:49:41 <ltomasbo> xD
14:49:45 <ltomasbo> I don't know...
14:50:30 <celebdor> we'll ask dmellado when he's available :-)
14:50:34 <irenab> I think its bug in the schedule, let;s give dmellado a chance to fix it
14:50:53 <celebdor> othewise we'll do bug fixing sessions
14:51:03 <irenab> +1
14:51:09 <irenab> any other topics?
14:51:41 <celebdor> not here
14:51:44 <irenab> then I guess we can close the meeting
14:51:48 <yboaron> Folks , please review #link https://review.openstack.org/#/c/536387/
14:52:05 <kaliya> garyloug: updates?
14:52:05 <irenab> yboaron, will do
14:52:12 <yboaron> irenab, 10x
14:52:31 <kaliya> also Danil told that he fixed his patches for interaction with the CNI-daemon, they can be reviewed
14:52:45 <kaliya> #link https://review.openstack.org/#/c/471012/ https://review.openstack.org/#/c/512280/ https://review.openstack.org/#/c/512281/
14:52:51 <irenab> kaliya, did you try them?
14:53:08 <kaliya> Not yet, but I plan to check on devstack the one which fails in zuul
14:53:20 <irenab> kaliya, great, thanks
14:53:24 <kaliya> This one https://review.openstack.org/#/c/524590/
14:53:31 <kaliya> if anyone has ideas, welcome :)
14:53:51 <garyloug> Hi kaliya, no update, something came up in my work last week that needed my attention. I'm working on the code right now and need to get it through internal review before I push.
14:54:14 <kaliya> we have a meeting tomorrow on this topic with garyloug and dmellado
14:54:21 <kaliya> don't forget! XD
14:54:32 <irenab> We plan to have a session on multivif at PTG also to go though the code, hope it will be the latest wehn we finalize the patches
14:54:39 <celebdor> kaliya: thanks
14:54:46 <kaliya> good to know irenab
14:55:05 <celebdor> dulek: after you finish the registry delete fix, take a look at danil's cni patches ;-)
14:55:07 <irenab> kaliya, worth to share the meeting details if someone wants to join
14:55:29 <celebdor> we should put the bluejeans links soon
14:55:30 <garyloug> Great, irenab has this session been scheduled for any day or time?
14:55:33 <kaliya> dmellado has the calendar event
14:55:42 <celebdor> garyloug: kaliya: right
14:55:47 <irenab> garyloug, I do not know
14:55:49 <dulek> celebdor: Sure, I'm helping daniel debug them. :)
14:55:50 <celebdor> he'll probably update tomorrow or wednesday
14:55:58 <celebdor> dulek: daniel or danil?
14:56:06 <garyloug> No worries, I'll keep an eye on the doc
14:56:07 <celebdor> we have both
14:56:09 <celebdor> :-)
14:56:11 <dulek> celebdor: danil, right.
14:56:14 <celebdor> :-)
14:56:24 <irenab> ok, we have 2 mins left
14:57:15 <irenab> thank you all for joining
14:57:21 <irenab> #endmeeting