| *** slaweq has joined #openstack-kuryr | 00:27 | |
| *** slaweq has quit IRC | 00:32 | |
| *** pmannidi has quit IRC | 00:37 | |
| *** pmannidi has joined #openstack-kuryr | 00:40 | |
| *** caowei has joined #openstack-kuryr | 00:53 | |
| *** jlabarre has quit IRC | 01:20 | |
| *** openstackgerrit has joined #openstack-kuryr | 01:28 | |
| openstackgerrit | Eunsoo Park proposed openstack/kuryr-kubernetes master: NestedMacvlanPodVIFDriver gets host port before request vif https://review.openstack.org/545760 | 01:28 |
|---|---|---|
| *** salv-orlando has joined #openstack-kuryr | 02:27 | |
| *** salv-orl_ has quit IRC | 02:30 | |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Add readiness and liveness checks to CNI. https://review.openstack.org/537721 | 02:49 |
| *** hongbin has joined #openstack-kuryr | 03:10 | |
| *** janki has joined #openstack-kuryr | 03:33 | |
| *** janonymous has joined #openstack-kuryr | 03:39 | |
| *** slaweq has joined #openstack-kuryr | 03:48 | |
| *** slaweq has quit IRC | 03:53 | |
| *** janki has quit IRC | 03:56 | |
| *** janki has joined #openstack-kuryr | 03:57 | |
| *** slaweq has joined #openstack-kuryr | 04:01 | |
| *** slaweq has quit IRC | 04:05 | |
| *** maysamacedos has quit IRC | 04:07 | |
| *** maysamacedos has joined #openstack-kuryr | 04:08 | |
| *** maysamacedos has quit IRC | 04:11 | |
| *** janki has quit IRC | 04:27 | |
| *** janki has joined #openstack-kuryr | 04:28 | |
| *** yamamoto has joined #openstack-kuryr | 04:34 | |
| *** slaweq has joined #openstack-kuryr | 04:38 | |
| *** slaweq has quit IRC | 04:43 | |
| *** caowei has quit IRC | 04:46 | |
| *** hongbin has quit IRC | 05:01 | |
| *** slaweq has joined #openstack-kuryr | 05:09 | |
| *** slaweq has quit IRC | 05:14 | |
| *** yboaron has joined #openstack-kuryr | 06:08 | |
| *** yboaron has quit IRC | 06:12 | |
| *** yboaron has joined #openstack-kuryr | 06:14 | |
| *** yboaron has quit IRC | 06:19 | |
| *** gcheresh_ has joined #openstack-kuryr | 06:19 | |
| *** yboaron has joined #openstack-kuryr | 06:19 | |
| *** yamamoto has quit IRC | 06:23 | |
| *** c00281451__ has quit IRC | 06:31 | |
| *** c00281451__ has joined #openstack-kuryr | 06:31 | |
| *** yamamoto has joined #openstack-kuryr | 06:33 | |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Add timeout to the wait_for method in plugin.sh https://review.openstack.org/545619 | 06:38 |
| *** juriarte has joined #openstack-kuryr | 07:09 | |
| *** threestrands has quit IRC | 07:10 | |
| *** janki has quit IRC | 07:20 | |
| *** vikasc has quit IRC | 07:33 | |
| *** pcaruana has joined #openstack-kuryr | 07:35 | |
| *** vikasc has joined #openstack-kuryr | 07:37 | |
| *** yamamoto has quit IRC | 07:42 | |
| *** yamamoto has joined #openstack-kuryr | 07:43 | |
| *** slaweq has joined #openstack-kuryr | 07:44 | |
| *** slaweq has quit IRC | 07:45 | |
| *** yamamoto has quit IRC | 07:46 | |
| *** Natanbro has joined #openstack-kuryr | 07:48 | |
| *** yamamoto has joined #openstack-kuryr | 07:50 | |
| *** slaweq has joined #openstack-kuryr | 07:54 | |
| *** yamamoto has quit IRC | 07:59 | |
| *** celebdor has quit IRC | 08:30 | |
| *** celebdor has joined #openstack-kuryr | 08:30 | |
| *** yamamoto has joined #openstack-kuryr | 08:31 | |
| *** c00281451__ has quit IRC | 08:45 | |
| *** c00281451__ has joined #openstack-kuryr | 08:45 | |
| *** yamamoto_ has joined #openstack-kuryr | 08:54 | |
| dulek | dmellado: Are you planning to split contents of .zuul.yaml into multiple .yaml in .zuul.d? That would address irenab concerns about your patch. | 08:58 |
| *** yamamoto has quit IRC | 08:58 | |
| dmellado | dulek: yep, IIRC I did leave some comment about that | 08:58 |
| dmellado | didn't I? | 08:58 |
| dmellado | I might do that as a follow-up patch, though | 08:59 |
| dulek | dmellado: I don't see a comment there. But anyway sure. We can make it a follow up, might be a bit easier to review. | 08:59 |
| dmellado | dulek: I did leave it | 09:00 |
| dmellado | see | 09:00 |
| dmellado | We can always create a .zuul.d directory and split the jobs in some files. i.e. | 09:00 |
| dmellado | zuul.d/kuryr_kubernetes.yaml | 09:00 |
| dmellado | zuul.d/01_kuryr_jobs.yaml | 09:00 |
| dmellado | zuul.d/02_kuryr_jobs-variants.yaml | 09:00 |
| dmellado | and so | 09:00 |
| dmellado | xD | 09:00 |
| dmellado | PS 24 | 09:00 |
| dulek | dmellado: Okay, sorry. :) | 09:00 |
| dmellado | I see that you replied afterwards with the same, no worries! | 09:00 |
| dmellado | so bsaically my current plan with the patch is add centos 7 experimental support | 09:01 |
| dmellado | then split | 09:01 |
| yboaron | Hi Folks , just confirmed that kuryr fails to create K8S service - please #link https://bugs.launchpad.net/kuryr-kubernetes/+bug/1750521 | 09:03 |
| openstack | Launchpad bug 1750521 in kuryr-kubernetes "K8S service resource creation - fails" [Undecided,New] | 09:03 |
| *** pmannidi has quit IRC | 09:16 | |
| dulek | yboaron: Awww, if it's caused by https://review.openstack.org/#/c/535548/ , we have it in stable/queens, don't we? | 09:20 |
| dulek | celebdor: That's another critical regression. ^ | 09:21 |
| dulek | :( | 09:21 |
| celebdor | let me check | 09:21 |
| celebdor | dulek: how does it cause it? | 09:21 |
| yboaron | dulek, yes it's critical for sure | 09:21 |
| dulek | celebdor: I'm trying to investigate that. | 09:21 |
| dulek | yboaron: What deployment type do you use? | 09:21 |
| yboaron | dulek, devstack + haproxy | 09:22 |
| dulek | yboaron: No kuryr-daemon, no containers? | 09:22 |
| yboaron | dulek, full details here #https://bugs.launchpad.net/kuryr-kubernetes/+bug/1750521 | 09:22 |
| openstack | Launchpad bug 1750521 in kuryr-kubernetes "K8S service resource creation - fails" [Critical,Confirmed] | 09:22 |
| yboaron | dulek, yes , non-containerized | 09:23 |
| dulek | yboaron: The log you provide is truncated. | 09:23 |
| *** garyloug has joined #openstack-kuryr | 09:24 | |
| yboaron | celebdor, I don't understand , why in case of exception (unhealthy set to True) , the LB handler is repeatedly called .. | 09:24 |
| dulek | yboaron: So I don't see lin numbers for example. | 09:24 |
| yboaron | dulek, let me check the full logs | 09:24 |
| dulek | yboaron: I'm more worried that health handler is called at all. You're not running kuryr-controller on K8s, so who's calling it? | 09:25 |
| dulek | yboaron: Ah, okay. It's just a report, not handler being called. | 09:26 |
| dulek | So basically an exception is raised by RetryHandler. It just seems that it's suppressed somewhere and we don't see it. | 09:26 |
| yboaron | dulek, I think that health handler is updating handler health (in retry.py) - and no one consume this information | 09:26 |
| dulek | So it might not be healthchecks code fault. Which is good. | 09:26 |
| dulek | yboaron: Mhm, that's okay. | 09:27 |
| yboaron | dulek, I think it's something related to exception handling , with the code we added in handlers (health handlers) - in case of exception in handler code - the event will repeatably sent to handler | 09:29 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Refactor Kuryr-Kubernetes gates to Zuul-V3 https://review.openstack.org/544959 | 09:29 |
| dulek | yboaron: I have a DevStack up and ready, so I'll work on debugging this. | 09:31 |
| celebdor | thanks dulek! | 09:33 |
| yboaron | dulek, good , BTW , for the service failure creation - solving the exception here - #link https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/handlers/lbaas.py#L275 | 09:33 |
| yboaron | dulek, def _has_pods(self, endpoints): | 09:34 |
| yboaron | ep_subsets = endpoints.get('subsets', []) | 09:34 |
| yboaron | if not ep_subsets: | 09:34 |
| yboaron | return False | 09:34 |
| yboaron | return any(True | 09:34 |
| yboaron | for subset in ep_subsets | 09:34 |
| yboaron | for address in subset.get('addresses', []) | 09:34 |
| yboaron | if address.get('targetRef', {}).get('kind') == 'Pod') | 09:34 |
| dulek | yboaron: You mean we magically started to get None as endpoints.get('subsets', [])? | 09:35 |
| dulek | yboaron: And that's the root cause? | 09:36 |
| yboaron | dulek, Don't think that this is the root cause | 09:36 |
| dulek | yboaron: Okay. I was starting to get confused. :) | 09:37 |
| dulek | yboaron: Okay, so… I've tried repeating what you've did and everything succeeded for me… | 09:39 |
| yboaron | dulek, deployed fresh VM/Devstack ? | 09:40 |
| dulek | yboaron: It's yesterday's master. | 09:40 |
| openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: [WIP] Add policy security-groups driver https://review.openstack.org/545872 | 09:40 |
| dulek | 519391fac094fe5e9a929998dd185fec3ef48af1 | 09:41 |
| yboaron | dulek, containerized ? | 09:41 |
| dulek | yboaron: No. | 09:41 |
| dulek | It has CNI daemon, but that shouldn't affect LBaaS codepaths. | 09:42 |
| yboaron | dulek, Hmmm... | 09:42 |
| dulek | yboaron: You're sure your neutron-lbaas isn't failing? | 09:42 |
| dulek | Maybe the log gets suppressed, but it's Neutron's fault? | 09:42 |
| yboaron | dulek, Don't think that it's neutron fault | 09:43 |
| yboaron | dulek, Let me reproduce it .. | 09:43 |
| celebdor | maybe yboaron has a nasty kubernetes version cooked just for him that gives him Nones | 09:44 |
| yboaron | celebdor, kube-yossi ?? | 09:45 |
| dulek | celebdor: IIUC that issue was unrelated. :P | 09:45 |
| celebdor | dulek: yboaron: I still don't understand that issue | 09:45 |
| celebdor | probably I didn't put enough time into that | 09:45 |
| celebdor | since I'm in ansible hell | 09:45 |
| dulek | Worst kind of hell. | 09:46 |
| dulek | Even worse than dependency hell. | 09:46 |
| celebdor | dulek: no. puppet would be that | 09:46 |
| celebdor | puppet + ruby | 09:46 |
| yboaron | dulek, I'm using the same kuryr version as you are - master (commit 519391fac094fe5e9a929998dd185fec3ef48af1) | 09:47 |
| dulek | celebdor: Please don't repeat that, my IRC client panicked and crashed. | 09:47 |
| dulek | yboaron: `sudo journalctl -u q-lbaasv2 --no-pager | grep ERROR` ? | 09:48 |
| yboaron | dulek, empty ... | 09:49 |
| dulek | Hm… | 09:51 |
| dulek | yboaron: Can you get the full log? | 09:53 |
| dulek | yboaron: I'll read again through that. | 09:54 |
| yboaron | dulek, Yes , in few minutes | 09:54 |
| dulek | Because I just tried again and LBaaS got created fine. | 09:54 |
| yboaron | dulek, Can I borrow your LbaaS ? :-) | 09:54 |
| dulek | yboaron: Uh, I don't know how to transfer one. ;) | 09:55 |
| yboaron | dulek, I need to pick something from post-office will ping U later | 10:01 |
| openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: [WIP] DisposableWatch for k8s client https://review.openstack.org/543448 | 10:10 |
| celebdor | dulek: I see you know the pain of ruby world too | 10:14 |
| dulek | celebdor: To be honest… Not much, but looks like my IRC client does. | 10:14 |
| celebdor | xD | 10:15 |
| dulek | celebdor: Oh, now I remember something… | 10:15 |
| celebdor | dulek: what do you use? | 10:15 |
| dulek | celebdor: 3 years ago I was told to install OSP. It was Foreman + Puppet back then. That took me 2 weeks of debugging Ruby code. | 10:15 |
| dulek | celebdor: For IRC? HexChat. I need to switch to WeeChat one day. | 10:16 |
| celebdor | weechat here | 10:16 |
| celebdor | :-) | 10:17 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Refactor Kuryr-Kubernetes gates to Zuul-V3 https://review.openstack.org/544959 | 10:30 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Refactor Kuryr-Kubernetes gates to Zuul-V3 https://review.openstack.org/544959 | 10:35 |
| dmellado | dulek: on the base jobs you're right | 10:37 |
| dmellado | they're not explicitly needed | 10:37 |
| dmellado | however, I was keeping them there as I was wondering if we could *not* trigger those but their parents with names like | 10:38 |
| dmellado | unit-test-kuryr-foo | 10:38 |
| dmellado | and so | 10:38 |
| dmellado | but let's see ;) | 10:38 |
| dulek | dmellado: Personally I prefer to be explicit, but I remember infra telling me not to do that. | 10:38 |
| dulek | So whatever, it's you dealing with them. :D | 10:38 |
| dmellado | lol | 10:38 |
| dmellado | well, I do plan to discuss that at on Mon | 10:39 |
| dmellado | but I was so sad when I saw that you didn't discuss the sessions on yesterday call! | 10:39 |
| dmellado | dulek: celebdor ltomasbo irenab | 10:39 |
| * dmellado sighs xD | 10:39 | |
| dmellado | I'll add them as I see fit | 10:39 |
| dulek | dmellado: We did. We've noticed this nap time from 15:30 to 16:00. | 10:39 |
| ltomasbo | xD | 10:40 |
| dmellado | 'nap time'? | 10:40 |
| dmellado | let me check | 10:40 |
| celebdor | dmellado: we were told you'd put a proposal | 10:40 |
| dulek | dmellado: I think it's easier to discuss explicit example. So fit them as you like and someone will complain if it's bad for that someone. | 10:40 |
| celebdor | we were ready to criticize it to hell | 10:40 |
| celebdor | but to come up with one | 10:41 |
| dmellado | lol | 10:41 |
| celebdor | is a bit much for a weekly irc meeting | 10:41 |
| dmellado | so, my plan was to leave 30' in between sessions | 10:41 |
| dmellado | as they tend to grow longer than intended | 10:41 |
| dmellado | i.e. first one in the morning | 10:41 |
| celebdor | cigarrete break? | 10:41 |
| dmellado | 9-10:30 then 11:12:30 | 10:41 |
| dmellado | or toilet | 10:41 |
| dulek | dmellado: Not a bad idea at all. | 10:41 |
| dmellado | or 'I need to get outta here' break | 10:41 |
| dmellado | thus the break | 10:42 |
| dmellado | so basically we have 2 slots in the morning and 2 in the afternoon | 10:42 |
| dmellado | and I'm pretty sure those 30' in between will be shorter xD | 10:42 |
| dulek | dmellado: In Cinder we had oh-no-they-are-discussing-taskflow-again breaks. :D | 10:42 |
| dmellado | lol | 10:42 |
| dmellado | besides that we've the infra session on Mon and the team photo on Tue, but that would only take 10 ' or so | 10:43 |
| dmellado | seems that we'll be getting on the field of croke park or something | 10:43 |
| dmellado | so I hope it won't be raining | 10:43 |
| kaliya | dmellado: celebdor do you have handy a line for consuming a kuryr-kubernetes patch for devstack? I'm trying this but no success https://ghostbin.com/paste/m7rfc | 10:46 |
| dmellado | kaliya: did you check our local.conf.sample files? | 10:46 |
| celebdor | kaliya: I think it's about putting the ref after the enable plugin | 10:47 |
| dmellado | https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/local.conf.sample | 10:47 |
| celebdor | as the last param | 10:47 |
| dulek | dmellado: Judging from my luck with Madrid, it'll be snowing. | 10:47 |
| dmellado | ah, the patch you mean | 10:47 |
| dmellado | yeah, just put up the gitref after the enable plugin | 10:48 |
| kaliya | with git.openstack.org or review.openstack.org | 10:48 |
| dmellado | [[local|localrc]] | 10:48 |
| dmellado | enable_plugin <NAME> <GITURL> [GITREF] | 10:48 |
| dmellado | name - an arbitrary name. (ex: glusterfs, docker, zaqar, congress) | 10:48 |
| dmellado | giturl - a valid git url that can be cloned | 10:48 |
| dmellado | gitref - an optional git ref (branch / ref / tag) that will be cloned. Defaults to master. | 10:48 |
| kaliya | now I try thanks | 10:49 |
| dmellado | kaliya: with git.opentack.org ;) | 10:49 |
| dmellado | kaliya: another way, if you already have an environment | 10:49 |
| dmellado | is just to use git-review -d my_patch | 10:49 |
| dmellado | and RECLONE=False when you go and restack | 10:50 |
| kaliya | good to know, ty | 10:51 |
| dmellado | kaliya: yw! | 10:52 |
| dmellado | dulek: don't bring the bad weather along with you xD | 10:52 |
| dulek | dmellado: It's easy to say, harder to do. :( | 11:00 |
| *** maysamacedos has joined #openstack-kuryr | 11:13 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Refactor Kuryr-Kubernetes gates to Zuul-V3 https://review.openstack.org/544959 | 11:21 |
| *** aojea has quit IRC | 11:46 | |
| yboaron | dulek, celebdor : Do you know in what order the following function in condition will be called #link https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/handlers/lbaas.py#L260 | 11:58 |
| yboaron | dulek, I've updated the logs in the bug | 11:59 |
| celebdor | yboaron: what do you mean? | 11:59 |
| yboaron | celebdor, I mean since no need to check all condition for False in that case | 11:59 |
| yboaron | celebdor, the function that causes the exception is _has_pods | 12:00 |
| yboaron | celebdor, that may explain why I got the exception and dulek Not | 12:00 |
| yboaron | celebdor, dulek : The subsets field is None in Endpoint spec until kubelet returns (VIF=Active) in our case | 12:02 |
| celebdor | yboaron: has_pods will be checked always when there is a lbaas_spec | 12:02 |
| yboaron | celebdor, OK, in that case dulek should also get the exception from has_pods function | 12:03 |
| *** aojea has joined #openstack-kuryr | 12:03 | |
| yboaron | celebdor, The health handler code add the following change : | 12:04 |
| yboaron | celebdor, #link https://review.openstack.org/#/c/535548/8/kuryr_kubernetes/handlers/retry.py | 12:04 |
| celebdor | yboaron: you don't need to put #link when we're not in a meeting ;-) | 12:04 |
| yboaron | celebdor, in case of exception I think will stay in forever loop . | 12:05 |
| yboaron | celebdor, isn't it ? | 12:05 |
| celebdor | yboaron: we should log the exception | 12:05 |
| celebdor | instead of just logging that we went unhealthy | 12:05 |
| celebdor | the exception content should be added to the log line | 12:05 |
| celebdor | also | 12:06 |
| * dulek finds all those Handler mixins so confusing to debug. | 12:06 | |
| yboaron | celebdor, but with current code will stay in forever loop with the same event , right ? | 12:06 |
| celebdor | I'm wondering whether we should reraise | 12:06 |
| celebdor | probably we should | 12:06 |
| dulek | Oh, I see now. We're reraising but then catching that one level above. :D | 12:07 |
| dulek | Okay, so it should be like that IMO. | 12:08 |
| yboaron | celebdor, dulek - in case of exception , it's an infinite loop with the same event right ? | 12:08 |
| dulek | yboaron: That's true. | 12:09 |
| dulek | Any exception actualy… | 12:09 |
| yboaron | dulek, celebdor - I still don't understand how you don't the exception in has_pods ??? do u have turbo kubelet-VIF ? | 12:10 |
| dulek | yboaron: Dunno really. Different K8s versions? | 12:11 |
| yboaron | stack@rrrrr kuryr-kubernetes]$ kubectl version | 12:11 |
| yboaron | Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"clean", BuildDate:"2017-12-07T16:05:18Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} | 12:11 |
| yboaron | Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"clean", BuildDate:"2017-12-07T16:05:18Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} | 12:11 |
| dulek | yboaron: Uhm, looks like I force 1.9.1. | 12:11 |
| yboaron | dulek, That's may explain it .. | 12:12 |
| dulek | :) | 12:12 |
| *** yamamoto_ has quit IRC | 12:13 | |
| yboaron | celebdor - I'll take of that bug , dulek I assume I'll need your help with the retry.py code .. | 12:14 |
| dulek | yboaron: I've already copied it and I'm trying to rewrite how it should look like. | 12:16 |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Add readiness and liveness checks to CNI. https://review.openstack.org/537721 | 12:16 |
| celebdor | yboaron: could you add the release note to the ext_subnet patch so we can approve it as well | 12:16 |
| celebdor | ? | 12:16 |
| dulek | yboaron: Though it's not obvious… | 12:16 |
| yboaron | celebdor, how do I add a release note ? | 12:17 |
| celebdor | reno | 12:17 |
| yboaron | celebdor, OK , I'll google it | 12:18 |
| dulek | yboaron: `pip install reno` | 12:18 |
| dulek | yboaron: `reno new made-option-blah-not-required` | 12:18 |
| dulek | yboaron: And you edit the file it gives you adding correct info there. | 12:19 |
| yboaron | dulek, thanks !! | 12:19 |
| dulek | yboaron: I guess the sections you need to edit are upgrade (new option, change in old's option behavior) and fixes (the bugfix). | 12:20 |
| dulek | yboaron: Rest you should delete. | 12:20 |
| dulek | yboaron: You can build notes by `tox -e releasenotes`. Build is in releasenotes/build/html | 12:21 |
| yboaron | dulek, you"ll take care of the retry.py , I'll keep the has_pods fix in https://review.openstack.org/#/c/545270/ , right ? | 12:21 |
| dulek | yboaron: Okay, I'll send the patch, is that what you want? | 12:21 |
| dulek | yboaron: Just remember that you need to commit the file with the releasenote before building, otherwise it'll be missing. ;) | 12:22 |
| yboaron | dulek, 10x ! Yes I'm fine with that | 12:23 |
| dulek | Awesome. | 12:23 |
| *** aojea has quit IRC | 12:27 | |
| *** garyloug has quit IRC | 12:28 | |
| *** yamamoto has joined #openstack-kuryr | 12:28 | |
| *** yamamoto has quit IRC | 12:31 | |
| *** yamamoto has joined #openstack-kuryr | 12:31 | |
| *** yamamoto has quit IRC | 12:31 | |
| dulek | yboaron, celebdor: I think that we only need to add `raise` in the last `except Exception` and we'll be okay. The logic will be: | 12:35 |
| dulek | If got an exception from self._exception try again until we hit max timeout. | 12:36 |
| dulek | When timeout is reached mark Handler as unhealthy and reraise exception. | 12:36 |
| *** yamamoto has joined #openstack-kuryr | 12:36 | |
| dulek | When got an exception from outside of the self._exception list, mark handler as unhealthy and reraise the exception. | 12:36 |
| dulek | This sounds correct to me. | 12:36 |
| celebdor | dulek: yes, that's what I thought too | 12:37 |
| dulek | yboaron: I don't think we need another patch for a single line. Can you add that single change into yours? | 12:37 |
| celebdor | dulek: as long as the health checks won't fail to return unhealthy after that last raise | 12:37 |
| celebdor | I think that's what it should be | 12:37 |
| yboaron | dulek, NP , add 'raise' at the last exception | 12:39 |
| dulek | yboaron: Exactly after this line: https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/handlers/retry.py#L72 | 12:39 |
| yboaron | dulek, yes , that's exactly what I did to get in exception trace | 12:40 |
| dulek | yboaron: :) | 12:40 |
| *** jlabarre has joined #openstack-kuryr | 12:43 | |
| openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: [WIP] Add policy security-groups driver https://review.openstack.org/545872 | 12:51 |
| *** salv-orlando has quit IRC | 13:12 | |
| openstackgerrit | Yossi Boaron proposed openstack/kuryr-kubernetes master: Make ext subnet config optional https://review.openstack.org/545270 | 13:12 |
| *** salv-orlando has joined #openstack-kuryr | 13:12 | |
| *** salv-orlando has quit IRC | 13:17 | |
| *** janonymous has quit IRC | 13:18 | |
| *** garyloug has joined #openstack-kuryr | 13:23 | |
| celebdor | dulek: yboaron: have you checked if https://review.openstack.org/#/c/545270/6..7/kuryr_kubernetes/tests/unit/handlers/test_retry.py does not mess with the health check behavior? | 13:33 |
| openstackgerrit | Shachar Snapiri proposed openstack/kuryr-kubernetes master: Add timeout to the wait_for method in plugin.sh https://review.openstack.org/545619 | 13:33 |
| yboaron | celebdor, I didn't .... | 13:34 |
| dulek | yboaron: Yeah, this test is fine, it just needs to do assert_raises. :) | 13:35 |
| yboaron | dulek, and since we added the 'raise' , I thought it reasonable to eliminate it | 13:36 |
| dulek | yboaron: Let me read it again. :) | 13:37 |
| celebdor | check with maysamacedos as well! | 13:38 |
| dulek | yboaron: Well, the name of the test needs to be changed as well, but test checks a correct codepath - going directly into the `except Exception:` clause. | 13:39 |
| dulek | yboaron: It checks the exact case you've hit with the bug. | 13:40 |
| celebdor | dulek: my point is | 13:40 |
| dulek | yboaron: It just has wrong asserts. Such exception needs to be reraised. | 13:40 |
| celebdor | why is that test there | 13:40 |
| celebdor | is it there to make sure that we don't go out in exception because that would break the health check consistency? | 13:40 |
| dulek | celebdor: Ah, I get this. Why is test asserting the exception will be suppressed. | 13:40 |
| celebdor | I don't know | 13:40 |
| dulek | celebdor: https://github.com/openstack/kuryr-kubernetes/blob/c43128d1fa57be8a6989ca7a23469785ab705cdc/kuryr_kubernetes/controller/handlers/pipeline.py#L55-L58 | 13:42 |
| dulek | celebdor: This is done by "wrapping", not inheritance. So I don't think not catching the exception will mess with stuff. | 13:43 |
| dulek | But let's wait for maysamacedos to confirm. | 13:43 |
| maysamacedos | I believe it will not affect the health checks | 13:46 |
| maysamacedos | dulek: yboaron: celebdor: it will be marked as unhealthy, and then logged | 13:47 |
| celebdor | good | 13:47 |
| dulek | maysamacedos: Do you remember if there was reason for suppressing the exception? Or it was just a mistake? | 13:48 |
| openstackgerrit | Omer Anson proposed openstack/kuryr-kubernetes master: TrivialFix: Make timeout a parameter in wait_for https://review.openstack.org/546149 | 13:51 |
| maysamacedos | I thought it was good to also mark as unhealthy when any kind of exception raises | 13:51 |
| maysamacedos | dulek: since it does not retry when an Exception occurs, only when Kuryr exceptions are raised | 13:53 |
| maysamacedos | from what I saw | 13:53 |
| dulek | maysamacedos: Yes, but still the unexpected exception should be reraised, not suppressed. | 13:54 |
| dulek | Anyway sounds like our fix is safe. :) | 13:54 |
| maysamacedos | dulek: yes | 13:54 |
| dulek | :) | 13:55 |
| maysamacedos | :) | 13:55 |
| celebdor | dulek: yboaron: maysamacedos: did or did k8s not restart the unhealthy controller? | 13:55 |
| yboaron | celebdor, its non containerized | 13:56 |
| maysamacedos | celebdor: with what yboaron proposed it will restart | 13:56 |
| maysamacedos | celebdor: let me check | 13:57 |
| celebdor | ah | 13:57 |
| celebdor | very well | 13:57 |
| yboaron | celebdor, maysamacedos : maybe I'm missing something , for non containerized - only the health handler is active | 13:58 |
| yboaron | celebdor, maysamacedos : do we need to support controller health manager/restart also for non-containerized ? I guess not | 14:00 |
| maysamacedos | yboaron you're right | 14:00 |
| *** maysamacedos has quit IRC | 14:05 | |
| *** maysamacedos has joined #openstack-kuryr | 14:09 | |
| *** maysamacedos has quit IRC | 14:12 | |
| *** yamamoto has quit IRC | 14:12 | |
| *** maysamacedos has joined #openstack-kuryr | 14:18 | |
| *** kaliya has quit IRC | 14:31 | |
| *** yboaron has quit IRC | 14:37 | |
| *** kaliya has joined #openstack-kuryr | 14:38 | |
| kaliya | has someone prepared an agenda for the meeting in 20 mins? dmellado | 14:39 |
| *** salv-orlando has joined #openstack-kuryr | 14:40 | |
| celebdor | kaliya: I have not | 14:45 |
| celebdor | maysamacedos: dulek: does review.openstack.org work well for you? | 14:45 |
| celebdor | it's not loading for me | 14:45 |
| dulek | celebdor: You're asking because I've did like 10 reviews today? :D | 14:46 |
| dulek | celebdor: It's fine. | 14:46 |
| maysamacedos | celebdor: yes | 14:46 |
| celebdor | darn | 14:46 |
| celebdor | maybe my firefox needs restarting | 14:46 |
| celebdor | all the other sites work | 14:47 |
| dmellado | kaliya: neither did I. I assume garyloug had one ;) | 14:47 |
| kaliya | I am sketching one in case | 14:47 |
| dulek | What meeting? Should I be aware of it? | 14:47 |
| celebdor | dulek: dpdk discussion | 14:47 |
| celebdor | you can join if you want, but getting the bugfixes in is top prio now | 14:47 |
| celebdor | irenab: don't you happen to be around? | 14:48 |
| garyloug | yea I have one, at least stuff we want to talk about on our end but open to anything that anyone else wants to discuss | 14:48 |
| *** celebdor has quit IRC | 14:48 | |
| *** yamamoto has joined #openstack-kuryr | 14:53 | |
| *** salv-orlando has quit IRC | 14:55 | |
| *** salv-orlando has joined #openstack-kuryr | 14:56 | |
| openstackgerrit | Maysa de Macedo Souza proposed openstack/kuryr-kubernetes master: Add readiness and liveness checks to CNI. https://review.openstack.org/537721 | 14:59 |
| *** salv-orlando has quit IRC | 15:00 | |
| *** celebdor has joined #openstack-kuryr | 15:04 | |
| *** garyloug has quit IRC | 15:07 | |
| *** garyloug has joined #openstack-kuryr | 15:10 | |
| *** slaweq has quit IRC | 15:23 | |
| *** slaweq has joined #openstack-kuryr | 15:24 | |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: Make ext subnet config optional https://review.openstack.org/545270 | 15:26 |
| *** slaweq has quit IRC | 15:28 | |
| dulek | yboaron, celebdor, irenab: I've cherry-picked https://review.openstack.org/#/c/546182/ onto stable/queens. | 15:30 |
| *** maysamacedos has quit IRC | 15:34 | |
| *** Natanbro has quit IRC | 15:39 | |
| *** maysamacedos has joined #openstack-kuryr | 15:40 | |
| *** salv-orlando has joined #openstack-kuryr | 15:47 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Refactor Kuryr-Kubernetes gates to Zuul-V3 https://review.openstack.org/544959 | 15:53 |
| *** gcheresh_ has quit IRC | 15:58 | |
| dulek | celebdor, irenab: https://etherpad.openstack.org/p/kuryr-queens-release - I've created a release commit/bugs checklist. Please take a look if there is anything you want to include in Queens. | 16:01 |
| maysamacedos | Hey guys, have anyone ever had the following error: 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance' when setting up the undercloud env | 16:01 |
| dulek | dmellado: Whoops, I've missed the elected PTL on that. See my message above. :) | 16:01 |
| maysamacedos | with https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/local.conf.pod-in-vm.undercloud.sample | 16:01 |
| maysamacedos | ? | 16:01 |
| dulek | maysamacedos: Yuuup. So it's a VM scheduling issue. | 16:02 |
| dmellado | dulek: too bad, but no worries xD | 16:02 |
| dmellado | lemme ehck | 16:02 |
| dmellado | check | 16:02 |
| openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: [WIP] Annotate pods that selected by k8s label-selector https://review.openstack.org/545663 | 16:02 |
| dulek | maysamacedos: A bit complicated thing to debug. So first check out if there are errors in nova-compute logs. n-cpu is the DevStack alias for that service. | 16:02 |
| dmellado | dulek: no worries in any case, I'm Rocky on xD | 16:03 |
| dulek | dmellado: :) | 16:03 |
| dulek | celebdor: What are next steps with this: https://review.openstack.org/#/c/545363/ ? | 16:03 |
| celebdor | dulek: I don't know if yboaron moved with it | 16:06 |
| celebdor | so I opened it now to fix the unit tests | 16:06 |
| celebdor | and get it merged | 16:06 |
| celebdor | then we'll need another one that modifies the SGs as listener ports change | 16:06 |
| celebdor | :'( | 16:06 |
| maysamacedos | thanks dulek | 16:08 |
| dulek | celebdor: Both need to get in stable/queens? | 16:10 |
| dulek | maysamacedos: Just in case: https://docs.openstack.org/devstack/latest/systemd.html :) | 16:10 |
| celebdor | dulek: you got it | 16:10 |
| celebdor | :-) | 16:10 |
| celebdor | the one you linked to is actually essential | 16:10 |
| dulek | celebdor: I'm doing preemptive stuff now, so I can work on that for an hour or two. Can you outline what's need to be done? | 16:11 |
| celebdor | dulek: very well | 16:13 |
| celebdor | just one moment | 16:13 |
| *** hongbin has joined #openstack-kuryr | 16:14 | |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: [WIP] services: Set missing SGs for haproxy provider https://review.openstack.org/545363 | 16:15 |
| celebdor | dulek: fix the unit tests for ^^ | 16:15 |
| celebdor | and if you can run it with a baremetal devstack with native ovs firewall | 16:15 |
| celebdor | to see if it fixes it | 16:15 |
| dulek | celebdor: Native OVS firewall is the default? | 16:16 |
| celebdor | dulek: no idea | 16:21 |
| celebdor | let me check | 16:21 |
| *** pcaruana has quit IRC | 16:21 | |
| *** slaweq_ has joined #openstack-kuryr | 16:25 | |
| *** slaweq_ has quit IRC | 16:30 | |
| *** maysamacedos has quit IRC | 16:33 | |
| celebdor | dulek: no. It's not the default in our local.conf | 16:33 |
| celebdor | you need to add this [[post-config|/$Q_PLUGIN_CONF_FILE]] | 16:34 |
| celebdor | [securitygroup] | 16:34 |
| celebdor | firewall_driver = openvswitch | 16:34 |
| celebdor | in the bottom of the local.conf | 16:34 |
| dulek | celebdor: Got it! | 16:34 |
| *** yamamoto has quit IRC | 16:38 | |
| celebdor | ltomasbo: you can review https://review.openstack.org/#/c/545363/2 | 16:39 |
| celebdor | but dulek will be fixing its uts | 16:39 |
| *** garyloug_ has joined #openstack-kuryr | 16:39 | |
| *** garyloug has quit IRC | 16:39 | |
| ltomasbo | celebdor, sure | 16:40 |
| openstackgerrit | Eyal Leshem proposed openstack/kuryr-kubernetes master: [WIP] Add policy security-groups driver https://review.openstack.org/545872 | 16:43 |
| *** yamamoto has joined #openstack-kuryr | 16:45 | |
| *** juriarte has quit IRC | 16:48 | |
| *** yamamoto has quit IRC | 16:50 | |
| *** yamamoto has joined #openstack-kuryr | 16:55 | |
| *** yamamoto has quit IRC | 16:59 | |
| *** maysamacedos has joined #openstack-kuryr | 17:00 | |
| celebdor | dulek: how's that going? | 17:14 |
| dulek | celebdor: I have unit tests fixed and I'm working on verifying the fix itself. | 17:14 |
| celebdor | great | 17:15 |
| dulek | celebdor: TypeError: 'LBaaSLoadBalancer' object has no attribute '__getitem__' :D | 17:16 |
| dulek | celebdor: It's an object, I guess. | 17:16 |
| dulek | celebdor: Okay, looks like it works. At least from inside the pods, but that was the idea, right? | 17:29 |
| celebdor | dulek: that was the entire idea | 17:33 |
| celebdor | :-) | 17:33 |
| celebdor | dulek: so push the fix and let's get this merged :-) | 17:34 |
| dulek | celebdor: But with the old code it works as well. | 17:34 |
| celebdor | mmmm | 17:34 |
| dulek | celebdor: Because port gets default SG if I see correctly. | 17:34 |
| celebdor | dulek: oh, right | 17:34 |
| celebdor | that's devstack | 17:34 |
| celebdor | xD | 17:34 |
| celebdor | in order to properly verify we should put a different SG | 17:35 |
| celebdor | in fact, maybe we should have this patch change devstack default | 17:35 |
| celebdor | to create a pod_sg | 17:35 |
| celebdor | using default is naughty | 17:35 |
| dulek | celebdor: Hah, I doubt I'll have time to do that now. I'll fix up unit tests after dict-ovo change and add a revert (LBaaS needs to be deleted when we get an exception on SG setup, isn't it?). | 17:36 |
| dulek | celebdor: We won't merge this today anyway. | 17:36 |
| *** pcaruana has joined #openstack-kuryr | 17:36 | |
| celebdor | yes, do that | 17:39 |
| celebdor | dulek: fscking hell | 17:46 |
| dulek | celebdor: Uh? | 17:46 |
| celebdor | the pod dns requests are not being returned in openshift-ansible deployment | 17:46 |
| celebdor | /etc/resolv.conf points to the node | 17:46 |
| celebdor | and even if I open the SG for port 53 and it is received | 17:46 |
| celebdor | an answer is not sent | 17:47 |
| celebdor | wtf is this shit | 17:47 |
| *** slaweq has joined #openstack-kuryr | 17:52 | |
| *** yamamoto has joined #openstack-kuryr | 17:55 | |
| *** slaweq has quit IRC | 17:57 | |
| *** yamamoto has quit IRC | 18:01 | |
| *** openstackgerrit has quit IRC | 18:03 | |
| *** openstackgerrit has joined #openstack-kuryr | 18:13 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Services: Set missing SGs for haproxy provider https://review.openstack.org/545363 | 18:13 |
| dulek | celebdor: I'll verify that more diligently tomorrow morning, need to go now. | 18:13 |
| celebdor | thanks dulek | 18:14 |
| *** pcaruana has quit IRC | 18:16 | |
| *** slaweq has joined #openstack-kuryr | 18:18 | |
| *** slaweq has quit IRC | 18:23 | |
| *** garyloug_ has quit IRC | 18:41 | |
| *** maysamacedos has quit IRC | 18:50 | |
| *** yamamoto has joined #openstack-kuryr | 18:57 | |
| *** yamamoto has quit IRC | 19:02 | |
| *** gcheresh has joined #openstack-kuryr | 19:26 | |
| *** slaweq has joined #openstack-kuryr | 19:31 | |
| *** slaweq has quit IRC | 19:36 | |
| *** salv-orlando has quit IRC | 19:39 | |
| *** salv-orlando has joined #openstack-kuryr | 19:40 | |
| *** salv-orlando has quit IRC | 19:44 | |
| *** yboaron has joined #openstack-kuryr | 19:50 | |
| *** yamamoto has joined #openstack-kuryr | 19:58 | |
| *** yamamoto has quit IRC | 20:02 | |
| *** slaweq has joined #openstack-kuryr | 20:10 | |
| *** slaweq has quit IRC | 20:14 | |
| *** pcaruana has joined #openstack-kuryr | 20:40 | |
| *** salv-orlando has joined #openstack-kuryr | 20:40 | |
| *** salv-orlando has quit IRC | 20:45 | |
| *** dmellado has quit IRC | 20:58 | |
| *** yamamoto has joined #openstack-kuryr | 20:59 | |
| *** yboaron has quit IRC | 21:00 | |
| *** salv-orlando has joined #openstack-kuryr | 21:03 | |
| *** yamamoto has quit IRC | 21:03 | |
| *** gcheresh has quit IRC | 21:33 | |
| *** pcaruana has quit IRC | 21:37 | |
| *** aojea has joined #openstack-kuryr | 21:40 | |
| *** gcheresh has joined #openstack-kuryr | 21:42 | |
| *** aojea has quit IRC | 21:44 | |
| *** gcheresh has quit IRC | 21:46 | |
| *** slaweq_ has joined #openstack-kuryr | 21:52 | |
| *** slaweq_ has quit IRC | 21:57 | |
| *** yamamoto has joined #openstack-kuryr | 22:00 | |
| *** yamamoto has quit IRC | 22:04 | |
| *** dmellado has joined #openstack-kuryr | 22:18 | |
| *** threestrands has joined #openstack-kuryr | 22:23 | |
| *** yamamoto has joined #openstack-kuryr | 23:00 | |
| *** pmannidi has joined #openstack-kuryr | 23:03 | |
| *** openstackgerrit has quit IRC | 23:04 | |
| *** yamamoto has quit IRC | 23:06 | |
| *** maysamacedos has joined #openstack-kuryr | 23:16 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!