| *** yamamoto has joined #openstack-kuryr | 00:34 | |
| *** yamamoto has quit IRC | 00:39 | |
| *** kzaitsev_pi has joined #openstack-kuryr | 00:46 | |
| *** pc_m- has joined #openstack-kuryr | 00:49 | |
| *** pc_m has quit IRC | 00:51 | |
| *** kzaitsev1pi has quit IRC | 00:51 | |
| *** ltomasbo has quit IRC | 00:51 | |
| *** hongbin_ has joined #openstack-kuryr | 00:59 | |
| *** premsankar has quit IRC | 01:08 | |
| *** salv-orl_ has joined #openstack-kuryr | 01:20 | |
| *** salv-orlando has quit IRC | 01:23 | |
| *** yamamoto has joined #openstack-kuryr | 01:35 | |
| *** yamamoto has quit IRC | 01:41 | |
| *** yamamoto has joined #openstack-kuryr | 02:37 | |
| *** dougbtv_ has quit IRC | 02:38 | |
| *** yamamoto has quit IRC | 02:43 | |
| *** janki has joined #openstack-kuryr | 03:01 | |
| *** jchhatbar has joined #openstack-kuryr | 03:10 | |
| *** janki has quit IRC | 03:12 | |
| *** yamamoto has joined #openstack-kuryr | 03:39 | |
| *** yamamoto has quit IRC | 03:45 | |
| *** hongbin_ has quit IRC | 03:50 | |
| *** yamamoto has joined #openstack-kuryr | 03:52 | |
| *** janonymous has joined #openstack-kuryr | 04:58 | |
| *** gcheresh has joined #openstack-kuryr | 05:04 | |
| *** gcheresh has quit IRC | 05:13 | |
| *** gcheresh has joined #openstack-kuryr | 05:46 | |
| *** dangtrinhnt has joined #openstack-kuryr | 06:16 | |
| *** pcaruana has joined #openstack-kuryr | 06:28 | |
| *** dangtrinhnt has quit IRC | 06:37 | |
| *** jchhatbar is now known as janki | 07:07 | |
| *** gcheresh has quit IRC | 07:12 | |
| *** gcheresh has joined #openstack-kuryr | 07:14 | |
| *** kiennt26 has joined #openstack-kuryr | 07:24 | |
| *** gcheresh has quit IRC | 07:27 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Execute kuryr-cni as `docker exec` https://review.openstack.org/555040 | 07:28 |
|---|---|---|
| *** janonymous has quit IRC | 07:48 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add Kuryr pods logs to gate results https://review.openstack.org/557732 | 07:59 |
| *** ltomasbo has joined #openstack-kuryr | 08:01 | |
| ltomasbo | ping dmellado | 08:01 |
| dmellado | ltomasbo: poor bouncer | 08:02 |
| *** ltomasbo has quit IRC | 08:03 | |
| *** oanson has quit IRC | 08:12 | |
| *** lihi has quit IRC | 08:12 | |
| *** irenab has quit IRC | 08:12 | |
| *** oanson has joined #openstack-kuryr | 08:13 | |
| *** lihi has joined #openstack-kuryr | 08:13 | |
| *** irenab has joined #openstack-kuryr | 08:14 | |
| *** ltomasbo has joined #openstack-kuryr | 08:16 | |
| *** celebdor has joined #openstack-kuryr | 08:31 | |
| *** garyloug has joined #openstack-kuryr | 08:47 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add Kuryr pods logs to gate results https://review.openstack.org/557732 | 09:08 |
| *** snapiri has quit IRC | 09:33 | |
| dulek | dmellado: FINALLY: http://logs.openstack.org/32/557732/13/check/kuryr-kubernetes-tempest-lbaasv2/1d34dbb/controller/logs/kubernetes/ | 09:46 |
| dulek | dmellado: I'll clean it up after lunch and let's merge it, can we? | 09:46 |
| * dulek leaves for lunch. | 09:47 | |
| dmellado | dulek: \o/ sure, awesome | 09:48 |
| dmellado | enjoy it! xD | 09:48 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Fix the experimental multinode jobs https://review.openstack.org/558762 | 09:52 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Fix the experimental multinode jobs https://review.openstack.org/558762 | 09:54 |
| *** garyloug has quit IRC | 09:54 | |
| *** kiennt26 has quit IRC | 10:11 | |
| *** garyloug has joined #openstack-kuryr | 10:23 | |
| *** atoth has joined #openstack-kuryr | 11:49 | |
| *** rh-jelabarre has joined #openstack-kuryr | 11:58 | |
| *** dougbtv has joined #openstack-kuryr | 12:09 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Add Kuryr pods logs to gate results https://review.openstack.org/557732 | 12:15 |
| *** dougbtv_ has joined #openstack-kuryr | 12:56 | |
| *** dougbtv has quit IRC | 12:56 | |
| *** maysamacedos has joined #openstack-kuryr | 13:02 | |
| *** jistr is now known as jistr|mtg | 13:30 | |
| dulek | celebdor: https://review.openstack.org/#/c/557732 - can we get this in? Having that on master would make debugging those all gates a lot easier. | 13:41 |
| celebdor | dulek: done | 13:41 |
| dulek | celebdor: My life is a little bit better now. :) | 13:41 |
| celebdor | dulek: :-) | 13:42 |
| dulek | celebdor: Okay, so maybe this will ring a bell for you. | 13:51 |
| dulek | celebdor: http://logs.openstack.org/40/555040/17/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/20d49ea/controller/logs/screen-kuryr-daemon.txt.gz | 13:51 |
| dulek | celebdor: Basically what we can see is - `openssl` complains about no data received from server. | 13:51 |
| dulek | celebdor: Then the wget which I switched to use HTTP on purpose is complaining about the same. | 13:52 |
| dulek | celebdor: And most likely wget with HTTPS was complaining about receiving no data as well. | 13:52 |
| dulek | So the TCP connection is established, but server does not answer. | 13:52 |
| dulek | I don't see any sign of connection in kubernetes-api logs. This is expected as no HTTP connection was established, right? | 13:54 |
| *** hongbin has joined #openstack-kuryr | 13:54 | |
| dulek | Is it possible that somehow it's connecting to something else? | 13:54 |
| celebdor | mmm | 13:55 |
| dulek | celebdor: http://logs.openstack.org/40/555040/17/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/20d49ea/controller/logs/iptables.txt.gz - not much in iptables for 10.1.0.129. | 13:55 |
| celebdor | are we sure it connects to the https and not the http? | 13:55 |
| dulek | celebdor: I've now tested http, just because I was out of other ideas. | 13:56 |
| dulek | celebdor: But previously I was trying HTTPS. It was clear in -v --debug output that it's HTTPs. | 13:56 |
| celebdor | dulek: the answer from the server had ssl stuff? | 13:57 |
| celebdor | from a container? | 13:57 |
| dulek | celebdor: I don't think there's any answer: http://logs.openstack.org/40/555040/16/experimental/kuryr-kubernetes-tempest-daemon-containerized-lbaasv2/a4cfe31/controller/logs/screen-kuryr-daemon.txt.gz#_Apr_04_23_30_58_652389 | 13:58 |
| celebdor | dulek: that's a bit my point | 14:00 |
| celebdor | that for some reason either the api doesn't work or the lb doesn't work | 14:01 |
| dulek | celebdor: API does work for e.g. kubelet, right? | 14:01 |
| dulek | Or the controller (which uses host networking and connects to the API directly) | 14:01 |
| dulek | So it's LB? | 14:02 |
| dulek | I've tried longer timeouts, IIRC up to 10 minutes. | 14:02 |
| dulek | And the gate is lbaasv2, not Octavia. | 14:03 |
| celebdor | dulek: the kubelet does not use the LB | 14:05 |
| dulek | celebdor: Yup, so it should be LB. But why on the gate only? | 14:05 |
| celebdor | I don't know yet | 14:05 |
| dulek | celebdor: Well, I can get any command you want into cni_ds_init, so if you have an idea what can help us - shoot. | 14:08 |
| dulek | celebdor: Plus with the commit in merge queue I can add some outputs from the host as well if needed. | 14:08 |
| celebdor | dulek: I was just reviewing the devstack code | 14:08 |
| celebdor | dulek: are you trying with k8s or origin? | 14:09 |
| dulek | celebdor: It's K8s. Origin does not work with containerization in the gate as well. | 14:09 |
| dulek | celebdor: https://review.openstack.org/#/c/557313/ - this was supposed to add OpenShift + containterized Kuryr gate, but after fixing 3 bugs it's still failing for some unknown reason. | 14:10 |
| *** kzaitsev_pi has quit IRC | 14:14 | |
| celebdor | dulek: could it be that somehow kubernetes-api is not listening at 6443? | 14:14 |
| celebdor | maybe it changed to 8443 now | 14:14 |
| dulek | celebdor: Uh… I might have different hyperkube than in the gate. | 14:15 |
| dulek | celebdor: Should be fairly easy to check… | 14:15 |
| dulek | celebdor: listing is available at https://158.69.79.121:6443/swaggerapi | 14:16 |
| dulek | And: Serving insecurely on 0.0.0.0:8080 | 14:16 |
| dulek | We're actually using this insecure connection for kubelet and kuryr-controller - it's in config. | 14:16 |
| celebdor | I know we use the insecure for kubelet and controller :P | 14:17 |
| celebdor | maybe at some point the ssl cli config that we do in devstack went to hell | 14:17 |
| celebdor | but you say it works on your devstack depoloyment | 14:18 |
| celebdor | can you please check if you use a different hyperkube version? | 14:18 |
| dulek | celebdor: I'm checking such connection again now. | 14:18 |
| dulek | celebdor: Yeah, then I'll use hyperkube. | 14:18 |
| dulek | s/use/switch | 14:18 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Handle lack of certificate in DevStack https://review.openstack.org/558470 | 14:19 |
| *** kzaitsev_pi has joined #openstack-kuryr | 14:21 | |
| dulek | celebdor: Yep, on my env I can connect. | 14:21 |
| celebdor | dulek: version? | 14:21 |
| dulek | celebdor: Gate has 1.9.0, I have 1.9.1. I'll check 1.9.0, though it's still long shot. | 14:22 |
| dulek | Another difference is that I have CentOS and gate's running on Ubuntu. | 14:22 |
| *** jistr|mtg is now known as jistr | 14:26 | |
| maysamacedos | celebdor dulek ltomasbo: I’ve been thinking how to expose the pool manager so it can be used not only by DevStack deployments and a feel ideas were discussed: | 14:35 |
| maysamacedos | 1. Create a k8s service exposing a port and use k8s credentials to communicate with it | 14:35 |
| maysamacedos | 2. Annotate the nodes | 14:35 |
| maysamacedos | 3. Pools CRD (we don’t have this yet) | 14:35 |
| maysamacedos | 4. Integrate with Keystone auth and add the endpoint/IP/URL to Keystone catalog. | 14:35 |
| maysamacedos | celebdor dulek ltomasbo: what do you think about? | 14:37 |
| maysamacedos | If we want to stick more to the k8s style I'm more inclined to the first option | 14:38 |
| dulek | Oh yes, this is a nice one. Initially I was proposing Keystone endpoint, but I expect celebdor to be furious about that idea. :D | 14:38 |
| dulek | K8s service has this disadvantage that it cannot really work with non-containerized Kuryr. | 14:40 |
| dulek | Annotating nodes to put an URL… Sounds pretty bad. Same goes with Pools CRDs. | 14:41 |
| *** pcaruana has quit IRC | 14:41 | |
| celebdor | dulek: develop "same goes with pools crds" | 14:45 |
| dulek | celebdor: Ah, communicating through pools CRDs editing is fine. Putting an URL for the manager there doesn't. Like what if I don't have any pool crated yet? | 14:47 |
| celebdor | ah | 14:49 |
| celebdor | ok | 14:49 |
| celebdor | now I get it | 14:49 |
| dulek | Can we put a dummy K8s Service that will be ignored by any handlers? | 14:50 |
| dulek | Like only to pass the info to anyone? | 14:50 |
| maysamacedos | dulek: I think we can. What is the case that this wouldn't be possible? | 14:53 |
| dulek | I just wonder if Kuryr won't start to try to wire it up. ;) | 14:53 |
| celebdor | dulek: it there's no endpoints we do nothing | 14:55 |
| celebdor | there's a kind of service called "external" | 14:55 |
| dulek | celebdor: Hm, okay, so that would work for us? Either expose containerized Kuryr through a Service or add a dummy service for non-containerized Kuryr deployments? | 14:57 |
| celebdor | or a mix | 15:00 |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: Add Kuryr pods logs to gate results https://review.openstack.org/557732 | 15:05 |
| dulek | celebdor: That's what I meant - both, depending on deployment type. | 15:09 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Execute kuryr-cni as `docker exec` https://review.openstack.org/555040 | 15:20 |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Do not merge https://review.openstack.org/558511 | 15:22 |
| celebdor | I was thinking that maybe one could fit both cases | 15:22 |
| celebdor | with a bit of tweaking | 15:22 |
| *** maysamacedos has quit IRC | 15:25 | |
| *** maysamacedos has joined #openstack-kuryr | 15:26 | |
| *** openstackgerrit has quit IRC | 15:34 | |
| maysamacedos | not sure i followed, how this would work celebdor? | 15:41 |
| *** pcaruana has joined #openstack-kuryr | 15:47 | |
| celebdor | maysamacedos: well. I was thinking that we could probably have an external service for both cases | 15:51 |
| maysamacedos | hmm ok | 15:58 |
| maysamacedos | Thank you dulek and celebdor | 16:04 |
| celebdor | maysamacedos: I also feel it's clunky | 16:08 |
| *** pcaruana has quit IRC | 16:16 | |
| *** phuoc has quit IRC | 16:23 | |
| maysamacedos | celebdor: by dummy service you mean creating a container that has a kind of proxy service for the pool manager? | 16:30 |
| dulek | maysamacedos: Rather creating a Service directly using K8s API. | 16:44 |
| dulek | maysamacedos: I'll see how this can be done tomorrow. | 16:44 |
| *** phuoc has joined #openstack-kuryr | 16:44 | |
| maysamacedos | ok, ty dulek | 16:51 |
| dulek | maysamacedos: https://kubernetes.io/docs/concepts/services-networking/service/ | 16:53 |
| dulek | maysamacedos: Search this for externalname. | 16:54 |
| dulek | I think this is what celebdor had in mind. | 16:54 |
| dulek | maysamacedos: One more resource: https://kubernetes.io/docs/tutorials/kubernetes-basics/expose-intro/ | 16:54 |
| *** openstackgerrit has joined #openstack-kuryr | 16:59 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Execute kuryr-cni as `docker exec` https://review.openstack.org/555040 | 16:59 |
| dulek | ltomasbo: It's CNI restarting? | 17:00 |
| dulek | ltomasbo: Shouldn't be affected, Secret is only attached to controller. | 17:01 |
| dulek | ltomasbo: Before submitting I've tested the patch… | 17:01 |
| *** ltomasbo has quit IRC | 17:01 | |
| *** hongbin has quit IRC | 17:07 | |
| *** hongbin has joined #openstack-kuryr | 17:18 | |
| *** janki has quit IRC | 17:19 | |
| *** garyloug has quit IRC | 17:45 | |
| *** pcaruana has joined #openstack-kuryr | 18:08 | |
| *** hongbin has quit IRC | 18:16 | |
| *** hongbin has joined #openstack-kuryr | 18:24 | |
| *** pcaruana has quit IRC | 18:47 | |
| *** salv-orl_ has quit IRC | 18:53 | |
| *** salv-orlando has joined #openstack-kuryr | 18:53 | |
| *** salv-orlando has quit IRC | 18:57 | |
| *** maysamacedos has quit IRC | 19:35 | |
| *** gcheresh has joined #openstack-kuryr | 19:41 | |
| *** yamamoto has quit IRC | 19:45 | |
| *** salv-orlando has joined #openstack-kuryr | 19:55 | |
| *** maysamacedos has joined #openstack-kuryr | 19:58 | |
| *** jistr has quit IRC | 20:18 | |
| *** jistr has joined #openstack-kuryr | 20:19 | |
| *** maysamacedos has quit IRC | 20:25 | |
| *** gcheresh has quit IRC | 20:43 | |
| *** yamamoto has joined #openstack-kuryr | 20:46 | |
| *** yamamoto has quit IRC | 20:52 | |
| *** yamamoto has joined #openstack-kuryr | 21:48 | |
| *** yamamoto has quit IRC | 21:53 | |
| *** maysamacedos has joined #openstack-kuryr | 22:15 | |
| *** salv-orlando has quit IRC | 22:19 | |
| *** salv-orlando has joined #openstack-kuryr | 22:19 | |
| *** jistr has quit IRC | 22:22 | |
| *** salv-orlando has quit IRC | 22:24 | |
| *** jistr has joined #openstack-kuryr | 22:36 | |
| *** maysamacedos has quit IRC | 22:37 | |
| *** yamamoto has joined #openstack-kuryr | 22:49 | |
| *** yamamoto has quit IRC | 22:55 | |
| *** threestrands has joined #openstack-kuryr | 23:02 | |
| *** threestrands has quit IRC | 23:02 | |
| *** threestrands has joined #openstack-kuryr | 23:02 | |
| *** hongbin has quit IRC | 23:05 | |
| *** salv-orlando has joined #openstack-kuryr | 23:20 | |
| *** salv-orlando has quit IRC | 23:24 | |
| *** celebdor has quit IRC | 23:32 | |
| *** yamamoto has joined #openstack-kuryr | 23:51 | |
| *** yamamoto has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!