| *** tonanhngo has joined #openstack-kuryr | 00:06 | |
| *** tonanhngo has quit IRC | 00:10 | |
| *** aojea has joined #openstack-kuryr | 00:14 | |
| *** tonanhngo has joined #openstack-kuryr | 00:15 | |
| *** aojea has quit IRC | 00:19 | |
| *** tonanhngo has quit IRC | 00:20 | |
| *** aojea has joined #openstack-kuryr | 00:24 | |
| *** aojea has quit IRC | 00:28 | |
| *** pmannidi has joined #openstack-kuryr | 00:30 | |
| *** c00281451 has joined #openstack-kuryr | 00:40 | |
| *** aojea has joined #openstack-kuryr | 00:42 | |
| *** aojea has quit IRC | 00:47 | |
| *** aojea has joined #openstack-kuryr | 00:51 | |
| *** aojea has quit IRC | 00:56 | |
| *** yamamoto has joined #openstack-kuryr | 00:56 | |
| *** aojea has joined #openstack-kuryr | 01:00 | |
| *** aojea has quit IRC | 01:04 | |
| *** yamamoto has quit IRC | 01:05 | |
| *** yamamoto has joined #openstack-kuryr | 01:06 | |
| *** aojea has joined #openstack-kuryr | 01:09 | |
| *** aojea has quit IRC | 01:14 | |
| *** yamamoto has quit IRC | 01:18 | |
| *** aojea has joined #openstack-kuryr | 01:18 | |
| *** yamamoto has joined #openstack-kuryr | 01:19 | |
| *** aojea has quit IRC | 01:23 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** kiennt has joined #openstack-kuryr | 01:25 | |
| *** aojea has joined #openstack-kuryr | 01:27 | |
| *** aojea has quit IRC | 01:32 | |
| *** yamamoto has joined #openstack-kuryr | 01:32 | |
| *** yamamoto has quit IRC | 01:35 | |
| *** yamamoto has joined #openstack-kuryr | 01:41 | |
| *** yamamoto has quit IRC | 01:44 | |
| *** caowei has joined #openstack-kuryr | 01:44 | |
| *** yamamoto has joined #openstack-kuryr | 01:45 | |
| *** yamamoto has quit IRC | 01:47 | |
| *** yamamoto has joined #openstack-kuryr | 01:49 | |
| *** yamamoto has quit IRC | 01:50 | |
| *** yamamoto has joined #openstack-kuryr | 01:52 | |
| *** yamamoto has quit IRC | 01:54 | |
| *** yamamoto has joined #openstack-kuryr | 02:13 | |
| *** yamamoto has quit IRC | 02:13 | |
| *** yamamoto has joined #openstack-kuryr | 02:22 | |
| *** yedongcan has joined #openstack-kuryr | 02:26 | |
| *** yamamoto has quit IRC | 02:29 | |
| *** yamamoto has joined #openstack-kuryr | 02:34 | |
| *** yamamoto has quit IRC | 02:36 | |
| *** yamamoto has joined #openstack-kuryr | 02:40 | |
| *** yamamoto has quit IRC | 02:41 | |
| *** s1061123 has quit IRC | 02:47 | |
| *** s1061123 has joined #openstack-kuryr | 03:08 | |
| *** caowei has quit IRC | 03:16 | |
| *** s1061123 has quit IRC | 03:27 | |
| *** s1061123 has joined #openstack-kuryr | 03:29 | |
| *** s1061123_ has joined #openstack-kuryr | 03:45 | |
| *** s1061123 has quit IRC | 03:45 | |
| *** yedongcan has quit IRC | 03:52 | |
| *** yedongcan has joined #openstack-kuryr | 03:52 | |
| *** yedongcan has left #openstack-kuryr | 03:53 | |
| *** s1061123 has joined #openstack-kuryr | 04:05 | |
| *** s1061123_ has quit IRC | 04:05 | |
| *** aojea has joined #openstack-kuryr | 04:20 | |
| *** aojea has quit IRC | 04:25 | |
| *** s1061123_ has joined #openstack-kuryr | 04:25 | |
| *** s1061123 has quit IRC | 04:26 | |
| *** aojea has joined #openstack-kuryr | 04:29 | |
| *** aojea has quit IRC | 04:34 | |
| *** aojea has joined #openstack-kuryr | 04:38 | |
| *** aojea has quit IRC | 04:42 | |
| *** aojea has joined #openstack-kuryr | 04:47 | |
| *** aojea has quit IRC | 04:52 | |
| *** aojea has joined #openstack-kuryr | 04:56 | |
| *** yamamoto has joined #openstack-kuryr | 04:59 | |
| *** aojea has quit IRC | 05:01 | |
| *** aojea has joined #openstack-kuryr | 05:05 | |
| *** aojea has quit IRC | 05:10 | |
| *** yboaron_ has joined #openstack-kuryr | 05:16 | |
| *** yamamoto has quit IRC | 05:31 | |
| *** gouthamr has quit IRC | 05:43 | |
| *** kiennt has quit IRC | 05:44 | |
| *** kiennt has joined #openstack-kuryr | 05:58 | |
| *** yamamoto has joined #openstack-kuryr | 06:05 | |
| *** yedongcan has joined #openstack-kuryr | 06:06 | |
| *** aojea has joined #openstack-kuryr | 06:09 | |
| *** aojea has quit IRC | 06:13 | |
| *** janki has joined #openstack-kuryr | 06:20 | |
| *** yamamoto has quit IRC | 06:23 | |
| *** aojea has joined #openstack-kuryr | 06:36 | |
| *** aojea has quit IRC | 06:41 | |
| *** vikasc has quit IRC | 06:42 | |
| *** aojea has joined #openstack-kuryr | 06:45 | |
| *** aojea has quit IRC | 06:51 | |
| *** vikasc has joined #openstack-kuryr | 06:54 | |
| *** yboaron_ has quit IRC | 07:07 | |
| *** aojea has joined #openstack-kuryr | 07:13 | |
| *** yamamoto has joined #openstack-kuryr | 07:16 | |
| *** aojea has quit IRC | 07:17 | |
| *** aojea has joined #openstack-kuryr | 07:21 | |
| *** yamamoto has quit IRC | 07:25 | |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: Skips kubernetes service without selector https://review.openstack.org/483891 | 07:28 |
|---|---|---|
| *** pmannidi has quit IRC | 07:29 | |
| *** egonzalez has joined #openstack-kuryr | 07:49 | |
| openstackgerrit | howard lee proposed openstack/kuryr master: Optimize link address https://review.openstack.org/485485 | 08:17 |
| *** garyloug has joined #openstack-kuryr | 08:19 | |
| *** neiljerram has joined #openstack-kuryr | 08:21 | |
| *** yboaron_ has joined #openstack-kuryr | 08:42 | |
| *** longfei_zhang has joined #openstack-kuryr | 08:42 | |
| longfei_zhang | irenab, i found one really strange thing, i start kuryr controller but after about 1 mins the controller is stopped watching from the log | 08:46 |
| longfei_zhang | the log shows : Stopped watching '/api/v1/pods' | 08:47 |
| longfei_zhang | Stopped watching '/api/v1/services' | 08:47 |
| longfei_zhang | don't know why | 08:48 |
| openstackgerrit | howard lee proposed openstack/fuxi master: Replace e.message with str(e) https://review.openstack.org/485497 | 08:48 |
| *** neiljerram has quit IRC | 09:00 | |
| longfei_zhang | kzaitsev_ws, do you know why? | 09:02 |
| longfei_zhang | here are the some error msg | 09:04 |
| longfei_zhang | http://paste.openstack.org/show/615985/ | 09:04 |
| longfei_zhang | it seems like the LB's issue | 09:23 |
| longfei_zhang | if i change the api_root address to one master's ip then no this issue | 09:23 |
| kzaitsev_ws | longfei_zhang: the error probably means that kuryr-k8s lost connection to k8s-api | 09:24 |
| kzaitsev_ws | it does not reconnect sadly | 09:24 |
| longfei_zhang | yes it seems like the lb's issue | 09:24 |
| longfei_zhang | i will check lb side | 09:25 |
| *** kzaitsev_ws has quit IRC | 09:36 | |
| *** kzaitsev_ws has joined #openstack-kuryr | 09:37 | |
| kzaitsev_ws | longfei_zhang: https://bugs.launchpad.net/kuryr-kubernetes/+bug/1705429 | 09:42 |
| openstack | Launchpad bug 1705429 in kuryr-kubernetes "kuryr-k8s does not reconnect to API in case of API restart" [Undecided,New] | 09:42 |
| kzaitsev_ws | sounds familiar? | 09:42 |
| apuimedo | kzaitsev_ws: darn. can you fill a bug about the reconnect? We had that functionality before the rewrite | 09:55 |
| kzaitsev_ws | apuimedo: bug/1705429 | 09:56 |
| kzaitsev_ws | dang | 09:56 |
| kzaitsev_ws | I forgot how bot works | 09:56 |
| kzaitsev_ws | #1705429 ? | 09:56 |
| kzaitsev_ws | T_T | 09:56 |
| kzaitsev_ws | apuimedo: https://bugs.launchpad.net/kuryr-kubernetes/+bug/1705429 | 09:56 |
| openstack | Launchpad bug 1705429 in kuryr-kubernetes "kuryr-k8s does not reconnect to API in case of API restart" [Undecided,New] | 09:56 |
| kzaitsev_ws | already | 09:56 |
| apuimedo | kzaitsev_ws: I'm a bit conflicted about how to address this | 09:57 |
| apuimedo | on one side, I think that this should be done in the health check | 09:57 |
| apuimedo | and have k8s restart the kuryr controller pod due to health | 09:57 |
| apuimedo | on the second side... | 09:57 |
| kzaitsev_ws | apuimedo: saw your discussions bout eventlet yesterday. what do you want to replace it with? | 09:58 |
| apuimedo | it may be safer to also try to reconnect from code a couple of times | 09:58 |
| apuimedo | kzaitsev_ws: I was looking at what octavia uses, cotyledon | 09:58 |
| apuimedo | but we wouldn't have threadgroups for that | 09:58 |
| apuimedo | so from how I see it there's four options | 09:58 |
| apuimedo | a) Implement our own threading (there's some libs for that) | 09:59 |
| kzaitsev_ws | A cotyledon is a significant part of the embryo within the seed of a plant | 09:59 |
| kzaitsev_ws | T_T | 09:59 |
| apuimedo | yeah | 09:59 |
| apuimedo | b) python3 asyncio again (but distros are not really ready for that it seems) | 09:59 |
| apuimedo | c) Try and solve the eventlet clusterfuck (but looking at requirements it doesn't really look that easy) | 10:00 |
| apuimedo | d) eventually move to golang | 10:00 |
| apuimedo | the saddes part... | 10:00 |
| apuimedo | All the options that occurred to me suck | 10:00 |
| janonymous | :D | 10:03 |
| kzaitsev_ws | yeah. I'm a bit afraid of even trying to look into eventlet situation, to see what's actually wrong there | 10:03 |
| kzaitsev_ws | (% | 10:03 |
| janonymous | apuimedo: option c) looks a lot of work to me just to find the exact cause which took my entire day today without moving a single step ahead :D | 10:05 |
| *** caowei has joined #openstack-kuryr | 10:05 | |
| kzaitsev_ws | apuimedo: about reconnecting. I'd say that I would expect the controller to attempt to reconnect and if that fails after X times blow the process up. | 10:06 |
| kzaitsev_ws | cause a controller that does not watch for Pod/Service events is essentially doing nothing and is confusing | 10:07 |
| apuimedo | kzaitsev_ws: afaik... There's all sorts of wrong | 10:07 |
| kzaitsev_ws | the tricky part is to make sure the CNI doesn't reconnect, since they share the code. but should be do-able | 10:07 |
| apuimedo | sorry, I meant to reply to janonymous | 10:07 |
| apuimedo | kzaitsev_ws: the CNI daemon sholud reconnect | 10:08 |
| kzaitsev_ws | hm | 10:08 |
| apuimedo | janonymous: kzaitsev_ws: I cheated a bit, there's one final option | 10:08 |
| * kzaitsev_ws thinking | 10:08 | |
| apuimedo | e) With the CNI split, have the pyroute2 interactions done in the calling cni executable | 10:09 |
| kzaitsev_ws | yeah, you're right. if the api disappears in the midst of a pod spawning the CNI should reconnect and wait for more events | 10:09 |
| apuimedo | so, in (e) flow, we'd have the following | 10:09 |
| apuimedo | Kubelet calls CNI executable | 10:09 |
| apuimedo | this one sends the info to CNI daemon | 10:10 |
| apuimedo | which gets the data from the pods api | 10:10 |
| apuimedo | returns it to the executable | 10:10 |
| apuimedo | and the executable does the pyroute2 stuff | 10:10 |
| janonymous | apuimedo: but that would not solve the purpose of split.. | 10:11 |
| apuimedo | janonymous: it would greatly reduce its purpose | 10:11 |
| janonymous | apuimedo: i.e to be lightweight binary... | 10:11 |
| apuimedo | by about half | 10:11 |
| apuimedo | that's why I originally did not mention it | 10:11 |
| apuimedo | f) have only the CNI daemon work with cotyledon | 10:12 |
| janonymous | apuimedo: maybe we can try for a few more days to find some solution around eventlet(might be waste of efforts too, but let us try :) ) | 10:13 |
| janonymous | apuimedo: f) part would loose backward compatibility.. | 10:14 |
| janonymous | apuimedo: no, maybe f) part would work...but donno what is right | 10:15 |
| apuimedo | (f) would be the less drastic change | 10:17 |
| apuimedo | it would mean less sharing of code between CNI and controller though | 10:17 |
| janonymous | apuimedo: how about using wsgi/ httpserver from eventlet... | 10:18 |
| janonymous | apuimedo: although patching do that already i guess.. | 10:19 |
| apuimedo | janonymous: you mean like keystone did? | 10:19 |
| janonymous | apuimedo: maybe.. | 10:19 |
| apuimedo | The difference is that we're not an API service, so running behing uwsgi/apache feels slightly odd | 10:19 |
| apuimedo | ok, more than slightly | 10:19 |
| janonymous | apuimedo: :D , right | 10:20 |
| janonymous | apuimedo: i will try to do patching for a few more time...will update if i get anything.. | 10:24 |
| apuimedo | janonymous: just so we don't repeat work if I get around to debug it, which approaches did you try? | 10:26 |
| janonymous | apuimedo: i tried with eventlet tracebacks to backtrack, sample tb: http://paste.openstack.org/show/615999/ | 10:29 |
| janonymous | apuimedo: i tried patching eventlet with different combinations of monkey patch(socket, thread,select) combinations | 10:30 |
| *** kiennt has quit IRC | 10:30 | |
| janonymous | apuimedo:moving out service out of eventlet and running calls of addnetwork to find normal beahaviour vs unexpected behaviour | 10:31 |
| janonymous | s/service out/service | 10:31 |
| janonymous | i.e to run service outside from eventlet dir | 10:31 |
| janonymous | daemon service | 10:31 |
| *** pc_m has quit IRC | 10:32 | |
| *** pc_m has joined #openstack-kuryr | 10:34 | |
| janonymous | apuimedo: EAGAIN is being raised mostly maybe due to timeout at some level but not sure + select call switching at oslo service level which uses eventlet i guess | 10:34 |
| janonymous | apuimedo: that's it | 10:34 |
| openstackgerrit | howard lee proposed openstack/kuryr master: Use https instead of http for docs links https://review.openstack.org/485485 | 10:37 |
| *** longfei_zhang has quit IRC | 10:39 | |
| *** pc_m has quit IRC | 11:16 | |
| *** pc_m has joined #openstack-kuryr | 11:20 | |
| *** atoth has joined #openstack-kuryr | 11:37 | |
| *** yedongcan has left #openstack-kuryr | 11:39 | |
| *** rwallner has joined #openstack-kuryr | 11:43 | |
| *** c00281451 is now known as zengchen | 11:46 | |
| *** mattmceuen has joined #openstack-kuryr | 12:02 | |
| *** yboaron__ has joined #openstack-kuryr | 12:29 | |
| *** yboaron_ has quit IRC | 12:32 | |
| *** mattmceuen has quit IRC | 12:40 | |
| apuimedo | kzaitsev_ws: vikasc: notice https://github.com/dcbw/kubernetes/commit/20ff9dce22733341f186719b9ba0b34b8fe71271 | 12:50 |
| apuimedo | this is dcbw's PoC for multi network CNI (he's a K8s dev) | 12:50 |
| apuimedo | see that they propose to give the plugin the path to a kubeconf file (with the certs and such) | 12:51 |
| *** vikasc has quit IRC | 12:54 | |
| *** longfei_zhang has joined #openstack-kuryr | 13:01 | |
| longfei_zhang | Hi apuimedo, in current watch logic, if the connection error happened then we will stop watching for everything | 13:03 |
| longfei_zhang | https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/k8s_client.py#L133 | 13:03 |
| longfei_zhang | https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/k8s_client.py#L140 will raise the exception but have not add the TODO | 13:04 |
| apuimedo | longfei_zhang: we should fix that asap | 13:04 |
| kzaitsev_ws | longfei_zhang: I've filead a bug, see ^^^ | 13:06 |
| apuimedo | not just in 140, any exception will | 13:06 |
| *** vikasc has joined #openstack-kuryr | 13:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/481186 | 13:16 |
| *** longfei_zhang has quit IRC | 13:16 | |
| *** gouthamr has joined #openstack-kuryr | 13:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/481186 | 13:32 |
| *** yamamoto has joined #openstack-kuryr | 13:39 | |
| *** mattmceuen has joined #openstack-kuryr | 13:48 | |
| *** yboaron__ has quit IRC | 13:49 | |
| *** yboaron__ has joined #openstack-kuryr | 13:49 | |
| *** longfei_zhang has joined #openstack-kuryr | 13:50 | |
| *** yamamoto has quit IRC | 13:50 | |
| *** longfei_zhang has quit IRC | 14:28 | |
| *** yamamoto has joined #openstack-kuryr | 14:37 | |
| *** janki has quit IRC | 14:45 | |
| *** janki has joined #openstack-kuryr | 14:45 | |
| *** janki has quit IRC | 14:47 | |
| *** yamamoto has quit IRC | 14:55 | |
| *** yamamoto has joined #openstack-kuryr | 14:57 | |
| *** yamamoto has quit IRC | 15:03 | |
| *** neiljerram has joined #openstack-kuryr | 15:58 | |
| *** yboaron__ has quit IRC | 15:59 | |
| *** rwallner has quit IRC | 15:59 | |
| *** rwallner has joined #openstack-kuryr | 16:01 | |
| *** egonzalez has quit IRC | 16:01 | |
| *** rwallner has quit IRC | 16:02 | |
| *** rwallner_ has joined #openstack-kuryr | 16:02 | |
| *** aojea has quit IRC | 16:04 | |
| *** neiljerram has quit IRC | 16:07 | |
| *** aojea has joined #openstack-kuryr | 16:10 | |
| *** aojea has quit IRC | 16:15 | |
| *** aojea has joined #openstack-kuryr | 16:29 | |
| *** aojea has quit IRC | 16:34 | |
| *** aojea has joined #openstack-kuryr | 16:57 | |
| *** tonanhngo has joined #openstack-kuryr | 16:57 | |
| *** rwallner_ has quit IRC | 17:01 | |
| *** aojea has quit IRC | 17:01 | |
| *** rwallner has joined #openstack-kuryr | 17:03 | |
| *** kzaitsev1pi has joined #openstack-kuryr | 19:25 | |
| *** kzaitsev_pi has quit IRC | 19:26 | |
| *** aojea has joined #openstack-kuryr | 19:34 | |
| *** aojea has quit IRC | 19:38 | |
| *** yboaron__ has joined #openstack-kuryr | 19:46 | |
| *** openstack has joined #openstack-kuryr | 19:57 | |
| *** aojea has joined #openstack-kuryr | 20:01 | |
| *** aojea has quit IRC | 20:05 | |
| *** aojea has joined #openstack-kuryr | 20:11 | |
| *** gouthamr has quit IRC | 20:14 | |
| *** aojea has quit IRC | 20:15 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** aojea has joined #openstack-kuryr | 20:20 | |
| *** aojea has quit IRC | 20:24 | |
| *** gouthamr has joined #openstack-kuryr | 20:28 | |
| *** aojea has joined #openstack-kuryr | 20:29 | |
| *** aojea has quit IRC | 20:33 | |
| *** aojea has joined #openstack-kuryr | 20:47 | |
| *** aojea has quit IRC | 20:52 | |
| *** aojea has joined #openstack-kuryr | 20:56 | |
| *** aojea has quit IRC | 21:01 | |
| *** gouthamr has quit IRC | 21:26 | |
| *** aojea has joined #openstack-kuryr | 21:34 | |
| *** aojea has quit IRC | 21:39 | |
| *** yboaron__ has quit IRC | 21:42 | |
| *** aojea has joined #openstack-kuryr | 21:43 | |
| *** rwallner has quit IRC | 21:48 | |
| *** aojea has quit IRC | 21:48 | |
| *** rwallner has joined #openstack-kuryr | 21:49 | |
| *** aojea has joined #openstack-kuryr | 21:53 | |
| *** aojea has quit IRC | 21:57 | |
| *** yamamoto has joined #openstack-kuryr | 21:57 | |
| *** rwallner has quit IRC | 21:59 | |
| *** yamamoto has quit IRC | 22:00 | |
| *** rwallner has joined #openstack-kuryr | 22:03 | |
| *** rwallner_ has joined #openstack-kuryr | 22:05 | |
| *** rwallner has quit IRC | 22:08 | |
| *** rwallner_ has quit IRC | 22:09 | |
| *** aojea has joined #openstack-kuryr | 22:13 | |
| *** aojea has quit IRC | 22:18 | |
| *** tonanhngo has quit IRC | 22:19 | |
| *** tonanhngo has joined #openstack-kuryr | 22:19 | |
| *** tonanhngo has quit IRC | 22:19 | |
| *** aojea has joined #openstack-kuryr | 22:22 | |
| *** tonanhngo has joined #openstack-kuryr | 22:26 | |
| *** aojea has quit IRC | 22:27 | |
| *** tonanhngo has quit IRC | 22:30 | |
| *** aojea has joined #openstack-kuryr | 22:31 | |
| *** aojea has quit IRC | 22:36 | |
| *** aojea has joined #openstack-kuryr | 22:40 | |
| *** aojea has quit IRC | 22:45 | |
| *** aojea has joined #openstack-kuryr | 22:49 | |
| *** aojea has quit IRC | 22:54 | |
| *** aojea has joined #openstack-kuryr | 22:59 | |
| *** yamamoto has joined #openstack-kuryr | 23:01 | |
| *** aojea has quit IRC | 23:03 | |
| *** yamamoto has quit IRC | 23:07 | |
| *** gouthamr has joined #openstack-kuryr | 23:08 | |
| *** garyloug has quit IRC | 23:15 | |
| *** aojea has joined #openstack-kuryr | 23:17 | |
| *** aojea has quit IRC | 23:21 | |
| *** aojea has joined #openstack-kuryr | 23:26 | |
| *** aojea has quit IRC | 23:30 | |
| *** aojea has joined #openstack-kuryr | 23:35 | |
| *** aojea has quit IRC | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!