| *** maysamacedos has quit IRC | 00:00 | |
| *** maysamacedos has joined #openstack-kuryr | 00:42 | |
| *** salv-orlando has quit IRC | 01:16 | |
| *** salv-orlando has joined #openstack-kuryr | 01:16 | |
| *** caowei has joined #openstack-kuryr | 01:27 | |
| *** hongbin has joined #openstack-kuryr | 01:27 | |
| *** janonymous has joined #openstack-kuryr | 03:22 | |
| *** janki has joined #openstack-kuryr | 03:53 | |
| *** hongbin has quit IRC | 04:05 | |
| *** jchhatbar has joined #openstack-kuryr | 04:30 | |
| *** janki has quit IRC | 04:33 | |
| *** maysamacedos has quit IRC | 05:04 | |
| *** yboaron has quit IRC | 05:38 | |
| *** natanbro has joined #openstack-kuryr | 05:58 | |
| *** snapiri- has quit IRC | 06:16 | |
| *** snapiri has joined #openstack-kuryr | 06:16 | |
| *** gcheresh has joined #openstack-kuryr | 06:32 | |
| *** pcaruana has joined #openstack-kuryr | 06:35 | |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Fix the experimental multinode jobs https://review.openstack.org/558762 | 06:41 |
|---|---|---|
| *** leyal has quit IRC | 06:42 | |
| *** lihi has quit IRC | 06:42 | |
| *** irenab has quit IRC | 06:42 | |
| *** oanson has quit IRC | 06:42 | |
| *** irenab has joined #openstack-kuryr | 06:43 | |
| *** oanson has joined #openstack-kuryr | 06:44 | |
| *** lihi has joined #openstack-kuryr | 06:45 | |
| *** lihi has quit IRC | 06:54 | |
| *** lihi has joined #openstack-kuryr | 06:55 | |
| *** lihi has quit IRC | 07:04 | |
| *** lihi has joined #openstack-kuryr | 07:05 | |
| *** natanbro has quit IRC | 07:05 | |
| *** yboaron has joined #openstack-kuryr | 07:14 | |
| *** yboaron has quit IRC | 07:15 | |
| *** lihi has quit IRC | 07:16 | |
| *** lihi has joined #openstack-kuryr | 07:17 | |
| *** lihi has quit IRC | 07:27 | |
| *** lihi has joined #openstack-kuryr | 07:28 | |
| *** jistr is now known as jistr|dentist | 07:34 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add documentation about supported kubernetes versions https://review.openstack.org/559342 | 07:35 |
| *** lihi has quit IRC | 07:40 | |
| *** lihi has joined #openstack-kuryr | 07:41 | |
| *** juriarte has quit IRC | 08:31 | |
| *** juriarte has joined #openstack-kuryr | 08:31 | |
| openstackgerrit | Merged openstack/kuryr-kubernetes master: [Trivial] Remove redundant check service is enabled https://review.openstack.org/557170 | 08:33 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: Fix the experimental multinode jobs https://review.openstack.org/558762 | 08:44 |
| *** garyloug has joined #openstack-kuryr | 08:45 | |
| dulek | celebdor: ping | 08:45 |
| *** jistr|dentist is now known as jistr | 09:02 | |
| *** juriarte has quit IRC | 09:06 | |
| *** juriarte has joined #openstack-kuryr | 09:07 | |
| *** juriarte has quit IRC | 09:08 | |
| *** juriarte has joined #openstack-kuryr | 09:08 | |
| dmellado | dulek: ping any news from the investigation? | 09:19 |
| dulek | dmellado: Well, I'm running out of my networking debugging skills. :P | 09:20 |
| dmellado | dulek: lol, so, overall? | 09:21 |
| dmellado | didn't you get to find anything unusual? | 09:21 |
| dulek | dmellado: I know the traffic gets only to Neutron router and isn't passed to K8s API. | 09:22 |
| dmellado | dulek: can you get to connect to the K8s API from cli in the gate? | 09:23 |
| dulek | dmellado: Sure. It's only the K8s API Service IP that's not working. | 09:23 |
| dmellado | hmmm | 09:29 |
| irenab | dmellado, dulek what are you investigating? | 09:40 |
| dulek | irenab: Gate failures on https://review.openstack.org/#/c/555040/ | 09:41 |
| irenab | multinode? | 09:42 |
| dulek | irenab: Containerized. For some reason there's no connectivity to K8s API from the pod. | 09:45 |
| celebdor | dulek: pong | 09:46 |
| dulek | celebdor: Can you help me with debugging this connectivity issue? | 09:47 |
| dulek | celebdor: I can add your key to the VM so you can login there. | 09:47 |
| celebdor | dulek: how may I help you | 09:47 |
| celebdor | ok | 09:47 |
| celebdor | dulek: I'm in | 09:49 |
| *** natanbro has joined #openstack-kuryr | 09:50 | |
| dulek | celebdor: It's https://10.1.0.129 that's not answering. | 09:50 |
| celebdor | yes, I saw it | 09:50 |
| celebdor | dulek: found it :D | 09:52 |
| dulek | celebdor: Well, it's good. So what's wrong? | 09:53 |
| celebdor | ah, no | 09:53 |
| celebdor | sorry, I forgot a param xD | 09:53 |
| *** jchhatbar has quit IRC | 09:55 | |
| irenab | dulek, in the kubelet log, there is : network: failed to find plugin "kuryr-cni" in path [/opt/kuryr-cni/bin /opt/stack/cni/bin] | 09:58 |
| dulek | Mhm, that's because cni_ds_init is failing. And it's failing because of connectivity issue with K8s API through LBaaS. | 10:00 |
| celebdor | irenab: that's because the cni container fails to find the API | 10:00 |
| celebdor | dulek: what I can see is that kubectl through the API LB works | 10:00 |
| celebdor | dulek: check .kube/config | 10:00 |
| celebdor | I changed it | 10:00 |
| celebdor | damn | 10:01 |
| celebdor | I thought I did | 10:01 |
| celebdor | let me do it again | 10:01 |
| dulek | :P | 10:01 |
| celebdor | the https one requires auth | 10:02 |
| celebdor | :P | 10:02 |
| celebdor | one sec | 10:02 |
| celebdor | it does reply though | 10:03 |
| dulek | Hm> | 10:04 |
| dulek | ? | 10:04 |
| *** livelace-link has quit IRC | 10:07 | |
| celebdor | dulek: I'm now trying to connect to it with certs | 10:09 |
| dulek | celebdor: The curl command had the certs supplied, so I think that's not the issue. | 10:10 |
| celebdor | I thought it only had the ca | 10:10 |
| celebdor | but I see that I get unable to connect to server EOF | 10:10 |
| dulek | celebdor: Yup, that's what I observed. | 10:12 |
| celebdor | ok, I at least got kubectl to work with the 6443 (without the lb) | 10:14 |
| celebdor | now I can concentrate on the LB issue | 10:14 |
| celebdor | dulek: I have a meeting now. I'll continue after that | 10:27 |
| dulek | celebdor: Okay, I'm going lunching in a moment. | 10:31 |
| celebdor | ok | 10:31 |
| dmellado | we should just remove tls-proxy and that's it xD | 10:41 |
| *** livelace-link has joined #openstack-kuryr | 10:45 | |
| *** livelace-link has quit IRC | 10:46 | |
| *** livelace-link has joined #openstack-kuryr | 10:48 | |
| *** livelace-link has quit IRC | 10:49 | |
| celebdor | dmellado: tls-proxy? | 11:14 |
| dmellado | yeah, dealing with some issues on the certs on multinode too | 11:15 |
| *** atoth has joined #openstack-kuryr | 11:44 | |
| *** gcheresh_ has joined #openstack-kuryr | 11:48 | |
| *** gcheresh has quit IRC | 11:48 | |
| *** gcheresh has joined #openstack-kuryr | 11:52 | |
| *** gcheresh_ has quit IRC | 11:53 | |
| *** maysamacedos has joined #openstack-kuryr | 11:53 | |
| dulek | dmellado: I tried that. Doesn't help with that issue. | 12:03 |
| openstackgerrit | Genadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Testing curl succeeds from pod to the service https://review.openstack.org/558482 | 12:41 |
| *** pcaruana has quit IRC | 12:46 | |
| *** pcaruana has joined #openstack-kuryr | 13:11 | |
| *** janki has joined #openstack-kuryr | 13:22 | |
| *** jchhatbar has joined #openstack-kuryr | 13:50 | |
| *** janki has quit IRC | 13:52 | |
| dulek | IRC meeting? I've already started it on #openstack-meeting-4. | 14:03 |
| celebdor | dulek: coming | 14:03 |
| *** celebdor1 has joined #openstack-kuryr | 14:09 | |
| *** celebdor1 is now known as apuimedo | 14:09 | |
| *** janonymous has quit IRC | 14:32 | |
| *** gcheresh has quit IRC | 14:40 | |
| apuimedo | dulek: I think I found it for real this time :P | 14:52 |
| dulek | apuimedo: Oh my, Christmas is early this year! | 14:52 |
| dulek | apuimedo: So what it is? | 14:53 |
| apuimedo | dulek: 2 minutes for verification of my hypothesis, hold on | 14:53 |
| apuimedo | here's the thing | 15:04 |
| apuimedo | first. We are putting the lbaas member IP wrong | 15:05 |
| apuimedo | we are putting the host IP and not the kubelet IP | 15:05 |
| apuimedo | second, since we generate the kubernetes api certs before we have the kubelet interface, the certificate does not include the kubelet ip address | 15:06 |
| dulek | apuimedo: Hm. Okay. So why does it work fine on my env? | 15:15 |
| apuimedo | dulek: is it nested? | 15:16 |
| dulek | apuimedo: Nope. | 15:17 |
| apuimedo | s/nested/pod-in-vm/ | 15:17 |
| dulek | apuimedo: It's just a VM with DevStack, pretty much like in the gate. | 15:17 |
| dulek | apuimedo: And why do we need kubelet IP as LBaaS member? | 15:18 |
| apuimedo | dulek: because the host IP is not part of the cloud | 15:20 |
| apuimedo | so the lbaas may not be able to reach it | 15:20 |
| apuimedo | curiously, now I can ping it from the LB | 15:21 |
| dulek | apuimedo: Are you changing it now? Because I see 10.1.0.68 as member IP. | 15:21 |
| apuimedo | dulek: yes, I'm testing my hypothesis | 15:21 |
| apuimedo | but for some reason if I go inside the LB and try to curl, it tells me no route to host | 15:22 |
| dulek | apuimedo: We needed to add a route to make 10.1… IPs reachable. Not sure if that's related. | 15:23 |
| apuimedo | dulek: let me show you | 15:24 |
| apuimedo | dulek: http://paste.openstack.org/show/718750/ | 15:24 |
| apuimedo | this is from inside the lb namespace, as you can see | 15:24 |
| dulek | apuimedo: That's odd. Ping routes okay, but not kubectl? | 15:25 |
| apuimedo | dulek: yeah | 15:25 |
| apuimedo | it totally looks like SG thing | 15:25 |
| apuimedo | but I added the missing sg | 15:26 |
| apuimedo | the kubelet did not have the SG of the (service_pod_access) | 15:26 |
| apuimedo | I'll check if it is SG | 15:27 |
| apuimedo | nope, modifying the default one didn't seem to work either | 15:28 |
| apuimedo | I'll try it from the router | 15:29 |
| apuimedo | dulek: :O | 15:30 |
| apuimedo | Can't get there either from the router | 15:30 |
| apuimedo | wtf | 15:30 |
| dulek | apuimedo: I've noticed that router port has no SG set, but I thought it's expected. | 15:31 |
| apuimedo | dulek: that's expected, yes | 15:31 |
| dulek | apuimedo: Any ideas what to do next? | 15:37 |
| apuimedo | dulek: of course | 15:38 |
| apuimedo | :-) | 15:38 |
| apuimedo | dulek: it's working now | 15:38 |
| apuimedo | :-) | 15:39 |
| apuimedo | try | 15:39 |
| apuimedo | kubectl --kubeconfig=/opt/stack/.kube/lbconfignoverify get nodes | 15:39 |
| dulek | apuimedo: Yep, it does. | 15:39 |
| apuimedo | dulek: also, if you have your env available | 15:39 |
| apuimedo | paste me the output of "sudo iptables -n -L -v" | 15:39 |
| dulek | apuimedo: I've unstacked, so it might not include DevStack's stuff. | 15:40 |
| apuimedo | dulek: it prolly doesn't | 15:40 |
| apuimedo | but let's check anyways | 15:40 |
| dulek | apuimedo: http://pastebin.test.redhat.com/573521 | 15:42 |
| apuimedo | it's definitely different | 15:43 |
| dulek | apuimedo: Okay, so how do we fix that? | 15:44 |
| apuimedo | dulek: now I'm checking the possibilities | 15:45 |
| apuimedo | I still believe we should do the fixes I put above | 15:45 |
| dulek | apuimedo: #1 - SG for loadbalancer port. | 15:46 |
| dulek | #2 - 10.1.0.68 as member of the LB. | 15:46 |
| dulek | #3 - certificate. | 15:46 |
| apuimedo | yup | 15:47 |
| apuimedo | and now we need to see why the rule I put on openstack-INPUT is necessary | 15:48 |
| apuimedo | cause it feels weird to have to put it | 15:48 |
| dulek | apuimedo: tcp dpt:6443 - this one? | 15:49 |
| apuimedo | sudo iptables -I openstack-INPUT 1 -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 -dport 6443 -j ACCEPT | 15:49 |
| apuimedo | this one | 15:49 |
| dulek | Don't you just manually adjusting an SG with that? | 15:50 |
| apuimedo | sorry, sudo iptables -I openstack-INPUT 1 -p tcp -s 0.0.0.0/0 -d 0.0.0.0/0 --dport 6443 -j ACCEPT | 15:50 |
| apuimedo | dulek: I don't think it is equivalent to an SG | 15:52 |
| apuimedo | dulek: if you remember this is using hybrid firewall | 15:52 |
| apuimedo | so our kubelet interface does not get any SG applied to it | 15:53 |
| apuimedo | I may be wrong, but I think the problem is that the communication does not go through 16M 4383M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 | 15:54 |
| apuimedo | well, of course it doesn't match that | 15:56 |
| apuimedo | the point is | 15:56 |
| apuimedo | communication from 10.1.0.129 reaches the host networking namespace | 15:56 |
| apuimedo | (from the LB vip, from its namespace) | 15:56 |
| apuimedo | and the host networking iptables rules are applied | 15:56 |
| apuimedo | since we have no rule for the k8s api server | 15:57 |
| apuimedo | it goes to hell | 15:57 |
| *** gianpietro has joined #openstack-kuryr | 15:57 | |
| dulek | apuimedo: Sounds like adding the rule is acceptable then? | 15:58 |
| apuimedo | dulek: I'm debating myself on that point :P | 15:58 |
| apuimedo | did you try with ovs firewall or with hybrid? | 15:59 |
| apuimedo | in your env | 15:59 |
| dulek | apuimedo: The default, so OVS IIRC. | 15:59 |
| apuimedo | dulek: I thought the default is hybrid | 16:00 |
| dulek | apuimedo: Ah, right, it is. | 16:01 |
| dulek | apuimedo: You want me to switch to OVS and recheck? | 16:01 |
| apuimedo | dulek: you probably guessed, but the question is whether this iptables rule is necessary with the native driver or if it is not | 16:02 |
| apuimedo | dulek: your env is centos? | 16:02 |
| dulek | apuimedo: Yup. | 16:03 |
| apuimedo | dulek: I don't know if it changed after you unstacked | 16:04 |
| apuimedo | but if you looked at your iptables, INPUT chains don't end up with reject | 16:04 |
| apuimedo | so that's the difference | 16:04 |
| *** gianpietro has quit IRC | 16:05 | |
| dulek | apuimedo: Right. Plus I don't have issues with the rest of stuff you've listed - IP or certificate. | 16:05 |
| dulek | apuimedo: Gotta go now, I'll produce the patch fixing SG, member IP and certificate later in the evening. I don't have an opinion on what to do with iptables rule though. | 16:08 |
| dulek | apuimedo: BTW - gate is configured with hybrid as well: `firewall_driver = iptables_hybrid` | 16:10 |
| *** jchhatbar has quit IRC | 16:10 | |
| apuimedo | dulek: IMHO, these gate iptables rules are probably part of the image they use in infra to prevent hacks | 16:12 |
| apuimedo | dulek: IMHO, you should add the iptable rule for now as I put it above | 16:12 |
| apuimedo | install it just after creating the kubelet device | 16:12 |
| apuimedo | we can discuss it in the patch review | 16:13 |
| dulek | apuimedo: Yup, I've found that place in the code. | 16:13 |
| apuimedo | dulek: thanks dulek | 16:13 |
| dulek | apuimedo: Okay, I'll add you to the review once finished. | 16:13 |
| apuimedo | I meant | 16:13 |
| apuimedo | dulek: Thanks Michał | 16:13 |
| dulek | apuimedo: It's me who owe you! I wouldn't be able to debug it myself. :) | 16:13 |
| apuimedo | dulek: I love these things | 16:14 |
| apuimedo | they are the most fun I have at work | 16:14 |
| apuimedo | :-) | 16:14 |
| *** shadower has quit IRC | 16:15 | |
| *** hongbin has joined #openstack-kuryr | 16:32 | |
| *** yamamoto has joined #openstack-kuryr | 16:37 | |
| *** yamamoto has quit IRC | 16:38 | |
| *** shadower has joined #openstack-kuryr | 16:44 | |
| *** garyloug has quit IRC | 16:49 | |
| *** yamamoto has joined #openstack-kuryr | 17:16 | |
| *** maysamacedos has quit IRC | 17:22 | |
| *** maysamacedos has joined #openstack-kuryr | 17:25 | |
| *** natanbro has quit IRC | 17:28 | |
| *** yamamoto has quit IRC | 17:57 | |
| *** gcheresh has joined #openstack-kuryr | 17:58 | |
| *** CrayZee has quit IRC | 18:12 | |
| *** atoth has quit IRC | 19:02 | |
| *** atoth has joined #openstack-kuryr | 19:04 | |
| *** atoth has quit IRC | 19:07 | |
| *** yamamoto has joined #openstack-kuryr | 19:18 | |
| *** gcheresh has quit IRC | 19:27 | |
| *** yamamoto has quit IRC | 19:33 | |
| *** yamamoto has joined #openstack-kuryr | 20:09 | |
| *** yamamoto has quit IRC | 20:18 | |
| *** yamamoto has joined #openstack-kuryr | 20:19 | |
| *** yamamoto has quit IRC | 20:22 | |
| *** salv-orlando has quit IRC | 20:36 | |
| *** pcaruana has quit IRC | 20:36 | |
| *** salv-orlando has joined #openstack-kuryr | 20:36 | |
| *** maysamacedos has quit IRC | 20:48 | |
| *** yamamoto has joined #openstack-kuryr | 20:54 | |
| *** yamamoto has quit IRC | 20:58 | |
| *** hongbin has quit IRC | 22:45 | |
| *** yamamoto has joined #openstack-kuryr | 23:22 | |
| *** yamamoto has quit IRC | 23:26 | |
| *** yamamoto has joined #openstack-kuryr | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!