| *** lakerzhou has quit IRC | 00:04 | |
| *** oanson has quit IRC | 00:08 | |
| *** oanson has joined #openstack-kuryr | 00:09 | |
| *** caowei has joined #openstack-kuryr | 00:53 | |
| *** lakerzhou has joined #openstack-kuryr | 00:58 | |
| *** lakerzhou1 has joined #openstack-kuryr | 01:01 | |
| *** lakerzhou has quit IRC | 01:04 | |
| *** kiennt has joined #openstack-kuryr | 01:08 | |
| *** hongbin has joined #openstack-kuryr | 01:56 | |
| *** wangbo has joined #openstack-kuryr | 02:07 | |
| *** hongbin has quit IRC | 02:43 | |
| *** hongbin has joined #openstack-kuryr | 02:49 | |
| *** robust has quit IRC | 03:10 | |
| *** lakerzhou1 has quit IRC | 03:33 | |
| *** hongbin has quit IRC | 03:51 | |
| *** kiennt has quit IRC | 03:53 | |
| *** janki has joined #openstack-kuryr | 04:32 | |
| *** wangbo has quit IRC | 04:35 | |
| *** caowei has quit IRC | 04:44 | |
| *** aojea has joined #openstack-kuryr | 04:48 | |
| *** aojea has quit IRC | 04:53 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/kuryr-kubernetes master: CNI Daemon template https://review.openstack.org/480028 | 04:54 |
|---|---|---|
| *** yamamoto has quit IRC | 04:57 | |
| *** wangbo has joined #openstack-kuryr | 05:04 | |
| janonymous | apuimedo: `/msg Chanserv OP #openstack-kuryr apuimedo` | 05:05 |
| janonymous | apuimedo: `/topic https://etherpad.openstack.org/p/kuryr-queens-vPTG ` | 05:05 |
| *** yamamoto has joined #openstack-kuryr | 05:07 | |
| *** caowei has joined #openstack-kuryr | 05:10 | |
| *** wangbo has quit IRC | 05:30 | |
| *** wangbo has joined #openstack-kuryr | 05:39 | |
| *** aojea has joined #openstack-kuryr | 05:46 | |
| *** aojea has quit IRC | 05:52 | |
| *** wangbo has quit IRC | 05:54 | |
| *** aojea has joined #openstack-kuryr | 05:56 | |
| *** aojea has quit IRC | 06:05 | |
| *** apuimedo has quit IRC | 06:05 | |
| *** huats_ has quit IRC | 06:05 | |
| *** apuimedo has joined #openstack-kuryr | 06:05 | |
| *** wangbo has joined #openstack-kuryr | 06:08 | |
| *** deepika08 has quit IRC | 06:08 | |
| *** aojea has joined #openstack-kuryr | 06:10 | |
| *** huats_ has joined #openstack-kuryr | 06:10 | |
| *** c00281451_ has quit IRC | 06:16 | |
| *** jchhatbar has joined #openstack-kuryr | 06:22 | |
| *** jchhatbar has quit IRC | 06:23 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid port update neutron call during pods boot up https://review.openstack.org/504915 | 06:23 |
| *** jchhatbar has joined #openstack-kuryr | 06:23 | |
| *** janki has quit IRC | 06:24 | |
| *** janki has joined #openstack-kuryr | 06:28 | |
| *** jchhatbar has quit IRC | 06:31 | |
| *** wangbo has quit IRC | 06:37 | |
| *** wangbo has joined #openstack-kuryr | 06:39 | |
| *** kiennt has joined #openstack-kuryr | 06:48 | |
| *** wangbo has quit IRC | 06:57 | |
| livelace2 | ltomasbo, Hello, BTW, is there any way to set port name as it does openshift for services (<svc>.<namespace>.cluster.local) Like <namespace>-<pod> ? | 06:58 |
| *** wangbo has joined #openstack-kuryr | 06:58 | |
| ltomasbo | hi livelace2 | 06:59 |
| livelace2 | it will be more convenient to recognize the ports | 06:59 |
| ltomasbo | we had the pod, and I guess we could add the namespace yes, but there is no option to enable that at the moment (at least that I know) | 06:59 |
| livelace2 | ok | 07:00 |
| ltomasbo | however, in the patch that I propose yesterday I enabled the option to not set any name at all | 07:01 |
| ltomasbo | as nova does when booting VMs | 07:01 |
| ltomasbo | and in order to speed up the booting process | 07:01 |
| livelace2 | It's cool too | 07:04 |
| livelace2 | ltomasbo, Could you answer me the question that I asked apuimedo "Do pods work well in your environment on one node"? | 07:09 |
| ltomasbo | livelace2, what do you mean in one node? | 07:10 |
| ltomasbo | in a deployment with just one server/VM? | 07:10 |
| ltomasbo | yep, what problem do you have? | 07:10 |
| ltomasbo | nested/baremetal? | 07:10 |
| livelace2 | I meant that there is one working node, where work two Pods, do they have any network connectivity problems ? | 07:11 |
| ltomasbo | not that I'm aware of | 07:12 |
| ltomasbo | are you using ml2/ovs>? | 07:12 |
| livelace2 | yes | 07:12 |
| ltomasbo | and the pods are nested (inside a VM) or baremetal? | 07:13 |
| ltomasbo | dulek, I'm testing your patches for containerized deployment and I'm getting this: http://paste.openstack.org/show/621390/ | 07:13 |
| ltomasbo | dulek, does it ring a bell? | 07:14 |
| ltomasbo | livelace2, and with network connectivity problem you mean that they cannot reach each other, or they cannot reach internet? | 07:15 |
| dulek | ltomasbo: It does. Are you using containers that you've built yourself? | 07:15 |
| ltomasbo | dulek, you mean for the cni and controller? | 07:16 |
| ltomasbo | I just used devstack to deploy them | 07:16 |
| *** deepika08 has joined #openstack-kuryr | 07:16 | |
| ltomasbo | with the KURYR_K8S_CONTAINERIZED_DEPLOYMENT=True | 07:17 |
| dulek | ltomasbo: Okay, so they should be built. I might made a mistake that didn't show up on my system, because it wasn't fresh. Let me check. | 07:17 |
| ltomasbo | dulek, I see this: | 07:17 |
| ltomasbo | ls /opt/stack/cni/bin/ | 07:17 |
| ltomasbo | loopback | 07:17 |
| ltomasbo | should there be only loopback there? | 07:18 |
| livelace2 | ltomasbo, Network completely | 07:18 |
| dulek | ltomasbo: Nope, looks like CNI installation didn't put it in there. | 07:18 |
| ltomasbo | livelace2, what if you create a VM? does it has connectivity? | 07:19 |
| dulek | ltomasbo: and "ls /etc/cni/net.d"? | 07:19 |
| ltomasbo | dulek, 10-kuryr.conf 99-loopback.conf | 07:19 |
| dulek | ltomasbo: Curious. Can you check out if kuryr-cni pod in kube-system namespace isn't failing? | 07:20 |
| dulek | ltomasbo: And if so - take a look on its logs. | 07:20 |
| ltomasbo | dulek, it says 0 restarts | 07:20 |
| ltomasbo | and describing the pod, it just says it started normally | 07:21 |
| dulek | ltomasbo: "ls /opt/cni/bin"? | 07:22 |
| ltomasbo | kuryr-cni kuryr-cni-bin | 07:22 |
| dulek | ltomasbo: Here those are! | 07:22 |
| ltomasbo | xD | 07:22 |
| *** pcaruana has joined #openstack-kuryr | 07:22 | |
| ltomasbo | where they should be? in /opt/stack/cni? | 07:22 |
| dulek | ltomasbo: Actually I wonder why your CNI binaries path is /opt/stack/cni. I thought that /opt/cni/bin is the standard. | 07:23 |
| dulek | ltomasbo: Setting those should help you: https://github.com/openstack/kuryr-kubernetes/blob/master/cni.Dockerfile#L10-L11 | 07:24 |
| ltomasbo | it seems it looks into /opt/kuryr-cni/bin instead of /opt/cni/bin | 07:25 |
| ltomasbo | dulek, ok, and then rebuild the cni, right? | 07:25 |
| dulek | ltomasbo: Yes. | 07:26 |
| ltomasbo | although those are already in the cni.Dockerfile.... | 07:26 |
| dulek | ltomasbo: Rather looks into /opt/stack/cni instead of /opt/cni/bin. Is /opt/stack/cni what's used in DevStack? If so we need to make DevStack build containers with changed option. | 07:26 |
| * dulek needs to check. | 07:27 | |
| ltomasbo | dulek, yep, probably it does | 07:27 |
| ltomasbo | dulek, thanks! I'll check too if it is just a matter of placing them in the right place! | 07:28 |
| ltomasbo | now I can play rebuilding them and learn a bit more than if it was working at the first attempt! | 07:28 |
| ltomasbo | xD | 07:28 |
| dulek | ltomasbo: Awesome. I'll analyze DevStack plugin to determine the correct path. Fix should be trivial. | 07:28 |
| ltomasbo | yep, I guess so! | 07:29 |
| *** janki is now known as janki|lunch | 07:45 | |
| ltomasbo | dulek, umm, I tried the next without success: | 07:49 |
| ltomasbo | ./tools/build_cni_daemonset_image | 07:49 |
| ltomasbo | kill the cni pod | 07:49 |
| ltomasbo | and create a new pod after that | 07:49 |
| ltomasbo | same error, looking at the wrong location | 07:49 |
| dulek | ltomasbo: You've changed the ENVs in Dockerfile, right? | 07:49 |
| ltomasbo | and now, I copy the kuryr-cni and kuryr-cni-bin to the /opt/stack/cni/bin | 07:49 |
| ltomasbo | and I'm getting this 23s 8s 3 kubelet, masterovn-aaa51502522f47e483ea0192acc5004722d2f306vm-0 Warning FailedSync Error syncing pod, skipping: failed to "KillPodSandbox" for "d8b1ef91-9d0e-11e7-ba79-fa163ef67899" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"demo-2293951457-dlfk2_default\" network: netplugin failed but error parsing its diagnostic | 07:50 |
| ltomasbo | message \"\": unexpected end of JSON input" | 07:50 |
| ltomasbo | dulek, no, I did not changed them | 07:50 |
| ltomasbo | you mean change them to opt/stack/cni?? | 07:50 |
| dulek | ltomasbo: You could, or add build-args to build command. Or add envs in kuryr-cni daemonset k8s manifest. :) | 07:51 |
| dulek | Anyway copying should help anyway. | 07:51 |
| dulek | And this JSON error is something new to me. Had you tried looking at kubelet logs? | 07:52 |
| apuimedo | livelace2: ltomasbo: it was this setup right https://ibb.co/fYC2B5 ? | 07:52 |
| *** egonzalez has joined #openstack-kuryr | 07:52 | |
| ltomasbo | yep, I wonder why it is not working with the copy | 07:52 |
| ltomasbo | maybe I copied after the cni was there, let me try again | 07:53 |
| livelace2 | apuimedo, Yes | 07:55 |
| ltomasbo | dulek, kubelet journal: http://paste.openstack.org/show/621393/ | 07:56 |
| apuimedo | livelace2: ltomasbo: so a21/ns and a22/ns can't talk to either 172.16.2.8 nor 172.16.2.9? | 07:57 |
| dulek | ltomasbo: Interestingly enough this isn't happening in the gate: http://logs.openstack.org/33/503733/7/check/gate-tempest-dsvm-lbaasv2-kuryr-kubernetes-ubuntu-xenial-nv/9f14dd1/logs/screen-kubelet.txt.gz | 07:58 |
| livelace2 | apuimedo, No, 2.9 is reachable, but 2.8 not. | 07:58 |
| dulek | ltomasbo: But if configs were put in wrong place, it gate might never be configured to use containerized CNI. Whoops. | 07:58 |
| apuimedo | livelace2: 2.7 and 2.5 are the addresses in the same subnet as 2.9 and 2.8? | 07:59 |
| ltomasbo | dulek, I don't really now, but in my case, event though config files where not in the proper place, the controller and cni were running as containers | 08:00 |
| ltomasbo | so, I guess the gate should have failed too | 08:00 |
| ltomasbo | perhaps I did something wrong when creating my devstack... | 08:00 |
| dulek | ltomasbo: Ah sure, but CNI container is just doing "sleep infinity". Until we have daemon, it's only purpose is to put binaries and configs in correct place. | 08:00 |
| ltomasbo | ahhh, true | 08:01 |
| dulek | ltomasbo: So if Kuryr was installed on the system and no CNI config was provided, then k8s might just use standard CNI. Controller definitely was running containerized. | 08:01 |
| livelace2 | apuimedo, Yes | 08:02 |
| dulek | I'm trying to dig the gate runs, but we don't have much configs copied there. | 08:02 |
| dulek | So I'll need to run this myself. | 08:02 |
| ltomasbo | dulek, I can give you access to my 'non-working' devstack deployment if that is of any help | 08:03 |
| dulek | ltomasbo: Let me see first if my patch sets the CNI_BIN_DIR and CNI_CONF_DIR correctly… | 08:04 |
| ltomasbo | ok, at least in my case the files where in those dirs | 08:05 |
| ltomasbo | but the cni was not looking for them there... | 08:05 |
| apuimedo | livelace2: what's the address of the VM that has the two namespaces? | 08:05 |
| dulek | ltomasbo: Yeah, I've wrote a patch that feeds correct envvars into docker build, just need to test it. | 08:06 |
| ltomasbo | dulek, ok great! | 08:06 |
| livelace2 | apuimedo, That VM is placed in another network, 172.16.1.0/24 | 08:07 |
| ltomasbo | dulek, https://review.openstack.org/#/c/490377/6/cni_builder | 08:07 |
| ltomasbo | could be that bin_dir_path is not defined...? | 08:08 |
| apuimedo | livelace2: cool | 08:08 |
| apuimedo | can you verify that there's the trunk ovs switch in the host for the central VM? | 08:09 |
| *** janki|lunch is now known as janki | 08:09 | |
| dulek | ltomasbo: Yes! Those are two different Docker images… | 08:10 |
| dulek | ltomasbo: cni.Dockerfile, where definitions were put isn't using cni_builder. So that's causing problems as well. | 08:11 |
| dulek | ltomasbo: Oh, this might cause the problem with JSON. Thank you! | 08:13 |
| dulek | ltomasbo: Can you do "cat /opt/cni/bin/kuryr-cni"? | 08:14 |
| ltomasbo | no problem! | 08:14 |
| ltomasbo | sure | 08:14 |
| ltomasbo | #!/bin/bash | 08:14 |
| ltomasbo | export PBR_VERSION='0.2.1.dev25' | 08:14 |
| ltomasbo | \/kuryr-cni-bin | 08:15 |
| ltomasbo | without \, but I cannot type it otherwise...:D | 08:15 |
| dulek | ltomasbo: Okay, so you can try putting correct path in there, right? I'm working on a fix. | 08:16 |
| ltomasbo | great! thanks dulek | 08:17 |
| livelace2 | apuimedo, absolutely, do you need output of commands ? | 08:21 |
| ltomasbo | dulek, even removing that and pointing to the right place: \/opt/cni/bin/kuryr-cni-bin | 08:22 |
| ltomasbo | I get the same error | 08:22 |
| ltomasbo | failed to find plugin \"kuryr-cni\" in path [/opt/stack/cni/bin /opt/kuryr-cni/bin]" | 08:22 |
| ltomasbo | it seems it tries to search not in /opt/cni/bin, but in /opt/kuryr-cni/bin | 08:23 |
| *** kiennt has quit IRC | 08:23 | |
| dulek | ltomasbo: Oh right, you need to copy them between directories as well. | 08:23 |
| dulek | ltomasbo: There are two bugs in one. :( | 08:23 |
| ltomasbo | dulek, xD | 08:23 |
| apuimedo | livelace2: yeah | 08:24 |
| ltomasbo | so, the solution is to have them in kuryr-cni? | 08:24 |
| ltomasbo | or make the cni point to /opt/cni? | 08:24 |
| dulek | ltomasbo: You need to place kuryr-cni and kuryr-cni-bin in correct place, which seems "/opt/stack/cni/bin". Then you need to edit kuryr-cni to point to /opt/stack/cni/bin/kuryr-cni-bin. | 08:26 |
| dulek | ltomasbo: This should be enough. | 08:26 |
| ltomasbo | dulek, ok! | 08:27 |
| ltomasbo | dulek, demo2-1575152709-6z9q0 1/1 Running | 08:29 |
| ltomasbo | yep, that is | 08:29 |
| dulek | ltomasbo: Thanks for testing and debugging! I'll come up with a fix soon. | 08:30 |
| ltomasbo | dulek, no problem! thanks for the help! it is really cool to have this running as containers! | 08:30 |
| apuimedo | dulek: please, remember to file a bug in launchpad | 08:31 |
| apuimedo | and reference it in the fix | 08:31 |
| dulek | apuimedo: Sure, | 08:34 |
| apuimedo | thanks dulek! | 08:34 |
| livelace2 | apuimedo, https://paste.fedoraproject.org/paste/75IZq-kZuyepojg5NsY8eA | 08:43 |
| ltomasbo | livelace2, note you are using the ovs-hybrid, instead of the ovs-firewall | 08:44 |
| ltomasbo | and trunk ports with ovs-hybrid and security groups are not working well | 08:44 |
| ltomasbo | so, either you move to ovs-firewall, or disable security groups on those ports | 08:45 |
| livelace2 | ltomasbo, https://paste.fedoraproject.org/paste/A2HA41jgw9zyASzNGXihCQ | 08:46 |
| livelace2 | sg permit all | 08:46 |
| apuimedo | livelace2: I'd still try with native ovs firewall | 08:47 |
| ltomasbo | did you restart the agent? | 08:47 |
| apuimedo | you need to reattach the ports | 08:47 |
| apuimedo | restarting the agent is not enough | 08:47 |
| ltomasbo | as in your previous paste-bin the output of the show port says it is ovs-hybrid | 08:47 |
| ltomasbo | ahh, ofc | 08:47 |
| ltomasbo | if the ports are already attached, you need to re-attach them | 08:48 |
| ltomasbo | in fact, probably, if you already have the vm running on that trunk port | 08:48 |
| ltomasbo | you will need to hard-reboot the vm | 08:48 |
| apuimedo | ltomasbo: no, you can hotunplug and hotplug IIRC | 08:50 |
| ltomasbo | really? | 08:51 |
| ltomasbo | there will be a qbr connecting the VM to the br-int, that should be drop, right? | 08:52 |
| apuimedo | ltomasbo: I think that's how I did it | 08:54 |
| apuimedo | but can't tell you for sure | 08:54 |
| ltomasbo | neither do I :( | 08:54 |
| livelace2 | Did new ports https://paste.fedoraproject.org/paste/jawbSZP6dZBC5CabO3SmTA | 08:58 |
| livelace2 | Reattaching doesn't help, only recreating | 08:58 |
| apuimedo | ok | 08:59 |
| livelace2 | Parent port needs the same, I suppose :( | 08:59 |
| apuimedo | livelace2: yup | 09:00 |
| apuimedo | create a new port | 09:00 |
| apuimedo | hot plug it | 09:00 |
| apuimedo | and move the subports there | 09:00 |
| *** yamamoto has quit IRC | 09:07 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Clean up ENV vars mistmatches in Dockerfiles https://review.openstack.org/505125 | 09:07 |
| dulek | ltomasbo: ^ It would be great if you could try this fix up (with removal of /opt/stack/cni before starting). | 09:09 |
| ltomasbo | dulek, sure, let me trash away my devstack deployment and generate one from scratch. I'll let you know as soon as it is up and running | 09:10 |
| dulek | ltomasbo: Awesome, thank you! | 09:10 |
| ltomasbo | dulek, np! | 09:10 |
| livelace2 | It seems to be ... working now :) | 09:13 |
| ltomasbo | livelace2, so, what was it? | 09:13 |
| livelace2 | https://paste.fedoraproject.org/paste/1uyc4Scr6yKRK5U5X2OYLQ | 09:13 |
| ltomasbo | the ovs-firewall? | 09:13 |
| ltomasbo | I mean, the ovs-firewall not being active? | 09:14 |
| apuimedo | great! | 09:15 |
| livelace2 | It was the fact, that I didn't recreate ports after changing of the firewall (hybrid vs native). With native all namespaces on one Node can ping other hosts in a network :) | 09:15 |
| livelace2 | I saw that on the hypervisor host iptables rules were disappeared and I thought that it's enough for firewall changing | 09:17 |
| apuimedo | aha | 09:17 |
| livelace2 | Anyway, now I can concentrate on kuryr itself. Thanks guys for your help :) | 09:17 |
| apuimedo | livelace2: thanks to you for the patient troubleshooting! | 09:17 |
| *** c00281451 has joined #openstack-kuryr | 09:17 | |
| *** c00281451 is now known as zengchen | 09:18 | |
| zengchen | ping apuimedo | 09:19 |
| apuimedo | zengchen: pong | 09:20 |
| zengchen | apuimedo:is the vPTG on 2-4 october | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Allow passing multiple VIFs to CNI https://review.openstack.org/471012 | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler https://review.openstack.org/462455 | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI https://review.openstack.org/462456 | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Allow requesting additional subnets via annotation https://review.openstack.org/482544 | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV documentation https://review.openstack.org/478458 | 09:21 |
| openstackgerrit | Danil Golov proposed openstack/kuryr-kubernetes master: [WIP] Allow setting specific ports for SRIOV handler https://review.openstack.org/478494 | 09:21 |
| *** robust has joined #openstack-kuryr | 09:25 | |
| zengchen | apuimedo:It does not say specific time for vPTG in the ethopad https://etherpad.openstack.org/p/kuryr-queens-vPTG | 09:26 |
| dulek | apuimedo: Good point, what timezone are we in? | 09:29 |
| dulek | During the VTG. | 09:29 |
| *** yamamoto has joined #openstack-kuryr | 09:32 | |
| *** robust has quit IRC | 09:32 | |
| apuimedo | I was planning on doing 13-15utc each day | 09:45 |
| janonymous | i am good with it! | 09:57 |
| *** livelace2 has quit IRC | 10:11 | |
| *** reedip has joined #openstack-kuryr | 10:13 | |
| *** alishaaneja has joined #openstack-kuryr | 10:17 | |
| *** wangbo has quit IRC | 10:27 | |
| *** yamamoto has quit IRC | 10:34 | |
| *** yamamoto has joined #openstack-kuryr | 10:38 | |
| ltomasbo | hi again dulek: I'm getting this if running with octavia http://paste.openstack.org/show/621407/ | 10:47 |
| ltomasbo | dulek, it seems it does not quite like the if that you included at the end of the local.conf :) | 10:47 |
| *** yamamoto has quit IRC | 10:47 | |
| dulek | ltomasbo: Uh-oh, right. | 10:48 |
| dulek | ltomasbo: I wonder how do you do that then… | 10:49 |
| ltomasbo | yep, I don't now either, I'll look for other examples, there may be some out there... | 10:50 |
| *** caowei has quit IRC | 10:50 | |
| ltomasbo | dulek, and if there is no way, perhaps we need 2 separate files, one for octavia, one for lbaas | 10:54 |
| dulek | ltomasbo: Okay, I think it may be done easier. | 10:58 |
| ltomasbo | probably... | 10:59 |
| dulek | ltomasbo: http://paste.openstack.org/show/621410/ - $OCTAVIA_CONF will be empty ($OCTAVIA_CONF_DIR/octavia.conf wasn't), so it will get ignored. | 10:59 |
| dulek | At least that's what I've found in DevStack docs. Let's see. | 11:00 |
| ltomasbo | :D | 11:00 |
| ltomasbo | sounds good | 11:01 |
| *** wangbo has joined #openstack-kuryr | 11:06 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add Pool Manager to handle subports https://review.openstack.org/498698 | 11:07 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add list and show pool commands to Pool Manager https://review.openstack.org/504410 | 11:07 |
| *** yamamoto has joined #openstack-kuryr | 11:23 | |
| *** robust has joined #openstack-kuryr | 11:30 | |
| *** wangbo has quit IRC | 11:44 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Fix local.conf.sample once again https://review.openstack.org/505187 | 11:44 |
| dulek | ltomasbo: ^ I've tried it and seems to be working. | 11:46 |
| dulek | Am I the only one not receiving emails from Gerrit? | 11:46 |
| *** wangbo has joined #openstack-kuryr | 11:46 | |
| *** robust has quit IRC | 11:46 | |
| ltomasbo | dulek, I'm testing it too, and yes, it seems to work | 11:46 |
| ltomasbo | dulek, there was a maintenance process yesterday, perhaps still some side effects | 11:47 |
| apuimedo | dulek: you sure you don't have the gerrit email still sending to intel? | 11:49 |
| alishaaneja | I am trying to setup kuryr-kubernetes with Devstack. I populated the contents of Devstack/local.conf with Kuryr-kubernetes/local.conf.sample and then ran ./stack.sh for Devstack. It gives me this error: https://gist.github.com/alisha17/282e31d4061e99d88057802884a30a5a . Can someone guide me how to resolve this? | 11:49 |
| apuimedo | alishaaneja: hey | 11:50 |
| apuimedo | alishaaneja: are you running this on a VM? | 11:50 |
| alishaaneja | Nope. should I setup devstack on say, Vagrant? | 11:51 |
| apuimedo | alishaaneja: using VMs is considered safer | 11:52 |
| apuimedo | :-) | 11:52 |
| dulek | apuimedo: 100% sure, I was receiving emails before yesterday's outage. | 12:02 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid port update neutron call during pods boot up https://review.openstack.org/504915 | 12:02 |
| apuimedo | weird | 12:03 |
| apuimedo | go to #openstack-infra then | 12:03 |
| *** robust has joined #openstack-kuryr | 12:19 | |
| deepika08 | Hey apuimedo: dulek ,team-kuryr ! :) | 12:24 |
| deepika08 | I am facing similar issue for Ubuntu 16.04 desktop in VM | 12:24 |
| deepika08 | , I tried finding some solutions , running as stack user , even tried conjure-up but no luck,ask.openstack .org showing can't be reached | 12:24 |
| *** atoth has joined #openstack-kuryr | 12:25 | |
| dulek | deepika08: Yup, ask.openstack.org seems down. Can you paste the error into paste.openstack.org and put the link here? | 12:36 |
| *** robust has quit IRC | 12:54 | |
| deepika08 | dulek : i lost the pastebin link produced earlier , need to install afresh , may take some time , i had mainly these two errors """ | 12:57 |
| deepika08 | generate-subunit 1505810377 253 fail | 12:57 |
| deepika08 | World dumping.../stack.sh:exit_trap:522 exit 100 | 12:57 |
| deepika08 | i will put the link asap! | 12:58 |
| dulek | deepika08: The real error should start somewhere before that. Those are only post-failure messages. | 12:58 |
| *** lakerzhou has joined #openstack-kuryr | 13:00 | |
| deepika08 | oh ! there was failure of Rabbit server before that | 13:00 |
| deepika08 | dulek^^ | 13:00 |
| *** robust has joined #openstack-kuryr | 13:06 | |
| dulek | deepika08: Hm. That's something unusual. Exact log could help, maybe there's networking problem and something is unable to connect to RabbitMQ. | 13:06 |
| *** gouthamr has joined #openstack-kuryr | 13:13 | |
| *** wangbo has quit IRC | 13:20 | |
| deepika08 | dulek https://paste.ubuntu.com/25572437/ | 13:29 |
| dulek | deepika08: journalctl -u rabbitmq-server | 13:30 |
| dulek | deepika08: This should show full rabbitmq log, which might be helpful. | 13:30 |
| dulek | deepika08: You probably need to run it with sudo. | 13:34 |
| dulek | janonymous, apuimedo: I have no luck with running CNI daemon (https://review.openstack.org/#/c/480028), pods are hanging in ContainerCreating state. | 13:35 |
| dulek | janonymous, apuimedo: Do you want me to fix this and work on finishing the patch? Judging from the TODO listed in commit message there's still a lot of work to do. | 13:36 |
| apuimedo | dulek: try your hand at it and update janonymous and I on the findings | 13:38 |
| dulek | apuimedo: Okay. I'm guessing the message is just lost somewhere between CNI and daemon, it shouldn't be too difficult to find where. | 13:39 |
| apuimedo | dulek: does CNI send the request to the socket file | 13:39 |
| dulek | apuimedo: I think so, need to check to be sure. First I'll try to unclog logging in daemon, I'll need it anyway. | 13:42 |
| apuimedo | cool | 13:42 |
| *** hongbin has joined #openstack-kuryr | 13:51 | |
| apuimedo | ltomasbo: the -1 is for visibility https://review.openstack.org/#/c/504915/3 | 13:57 |
| apuimedo | look at the question | 13:57 |
| ltomasbo | apuimedo, oohh true | 13:59 |
| ltomasbo | I forgot about that... | 13:59 |
| apuimedo | :-) | 14:00 |
| ltomasbo | I'll fix it asap | 14:00 |
| ltomasbo | other than that is ok? | 14:00 |
| *** robust has quit IRC | 14:02 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid port update neutron call during pods boot up https://review.openstack.org/504915 | 14:07 |
| *** wangbo has joined #openstack-kuryr | 14:08 | |
| apuimedo | ltomasbo: it is | 14:19 |
| apuimedo | +2 | 14:19 |
| ltomasbo | great! | 14:19 |
| ltomasbo | thanks! | 14:19 |
| ltomasbo | trying to fix the octavia l2 issue, at the end it has to be with the git version... | 14:20 |
| *** yamamoto has quit IRC | 14:20 | |
| apuimedo | xD | 14:20 |
| *** janki has quit IRC | 14:36 | |
| ltomasbo | dulek, did you stacking work at the end? (for the containerized version) | 14:43 |
| ltomasbo | in my case I see the kubernetes-scheduler and the kubernetes-controller-manager pods died during the process | 14:44 |
| dulek | ltomasbo: You mean https://review.openstack.org/#/c/505125/ ? I've used env to test local.conf.sample patch and forget about it. :P | 14:45 |
| dulek | ltomasbo: It happens to me sometimes, mostly due to etcd leader election failures. | 14:45 |
| ltomasbo | yep, I meant that one | 14:45 |
| dulek | ltomasbo: Normally restart of etcd and devstack@kube* help | 14:45 |
| dulek | s. | 14:45 |
| *** wangbo has quit IRC | 14:46 | |
| ltomasbo | ok, though it crashed during stacking, not sure if something else was missing | 14:46 |
| *** wangbo has joined #openstack-kuryr | 14:46 | |
| dulek | ltomasbo: I' | 14:49 |
| dulek | ltomasbo: I'll run it once other DevStack run finishes. | 14:49 |
| ltomasbo | ok | 14:49 |
| ltomasbo | I restarted those services | 14:50 |
| ltomasbo | and started the kubernetes controller and scheduler | 14:50 |
| ltomasbo | but still kuryr-cni crashes | 14:50 |
| dulek | ltomasbo: Same errors? | 14:51 |
| ltomasbo | no no, now the kuryr-cni pod is in error or crash status | 14:52 |
| ltomasbo | crashloopbackoff | 14:52 |
| ltomasbo | but probably the stacking crashed before finishing | 14:53 |
| *** wangbo has quit IRC | 14:55 | |
| *** wangbo has joined #openstack-kuryr | 14:56 | |
| *** wangbo has quit IRC | 14:58 | |
| *** alishaaneja has quit IRC | 15:06 | |
| apuimedo | ltomasbo: you tried the no port update patch already? | 15:16 |
| apuimedo | I'm considering trying it on my 2000 port scale lab | 15:16 |
| ltomasbo | yes, I tested it | 15:17 |
| ltomasbo | (with odl as backend) | 15:17 |
| ltomasbo | it is faster even in my small nested environemnt | 15:17 |
| ltomasbo | like 40% faster | 15:17 |
| apuimedo | ltomasbo: how did you check when the pools are loaded? | 15:17 |
| apuimedo | meh... I would have expected it to be 300% faster | 15:17 |
| ltomasbo | apuimedo, https://review.openstack.org/#/c/504410/ | 15:17 |
| apuimedo | but I will take 40% | 15:17 |
| apuimedo | xD | 15:17 |
| ltomasbo | I can check the status of the pools with that one | 15:17 |
| apuimedo | right | 15:18 |
| ltomasbo | curl --unix-socket /etc/kuryr/kuryr_manage.sock http://localhost/listPools -H "Content-Type: application/json" -X GET -d '{}' | 15:18 |
| ltomasbo | that tell you all the pools, and the amount of ports the have | 15:18 |
| ltomasbo | and with showPool, you get the information for a given pool | 15:18 |
| ltomasbo | other than that, perhaps it is good to include a LOG message when the process finish | 15:19 |
| ltomasbo | if you give me 5 minutes, I'll update https://review.openstack.org/#/c/504915/ | 15:19 |
| ltomasbo | apuimedo, what do you thing? | 15:19 |
| apuimedo | ltomasbo: a log message would be a good thing | 15:20 |
| ltomasbo | ok, I'll add it | 15:20 |
| *** yamamoto has joined #openstack-kuryr | 15:20 | |
| *** yamamoto has quit IRC | 15:25 | |
| *** dimak has quit IRC | 15:25 | |
| *** dimak has joined #openstack-kuryr | 15:26 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid port update neutron call during pods boot up https://review.openstack.org/504915 | 15:27 |
| ltomasbo | apuimedo, ^^ | 15:27 |
| ltomasbo | apuimedo, now there will be a message in kuryr-kubernetes when the ports are loaded into the pools | 15:28 |
| apuimedo | perfect | 15:33 |
| apuimedo | thanks ltomasbo | 15:33 |
| ltomasbo | np! | 15:33 |
| apuimedo | ltomasbo: somehow I'm not seeing the POOL updated log message | 15:41 |
| apuimedo | and I started it like 3 minutes ago | 15:41 |
| ltomasbo | umm | 15:42 |
| *** wangbo has joined #openstack-kuryr | 15:42 | |
| ltomasbo | apuimedo, how many ports? | 15:42 |
| apuimedo | 2k | 15:42 |
| ltomasbo | ok | 15:42 |
| apuimedo | but considering there's no pods | 15:42 |
| apuimedo | it shouldn't be this slow | 15:42 |
| apuimedo | should it? | 15:42 |
| ltomasbo | actually it will be faster if there are pods | 15:42 |
| ltomasbo | as you only need to get the information about the ones that are available | 15:43 |
| apuimedo | but this should only be a port list, shouldn't it? | 15:43 |
| apuimedo | or we do it per trunk? | 15:43 |
| ltomasbo | you need to get the information regarding the port | 15:43 |
| apuimedo | What I mean is | 15:43 |
| apuimedo | a single port list | 15:43 |
| apuimedo | contains the info about the parent port already | 15:43 |
| apuimedo | so we should be able to reconstruct the pools with that | 15:44 |
| apuimedo | shouldn't we? | 15:44 |
| ltomasbo | umm, let me see | 15:44 |
| ltomasbo | though that is already 2k calls to neutron | 15:44 |
| ltomasbo | the port list does not show the parent | 15:44 |
| ltomasbo | the port show do it | 15:44 |
| ltomasbo | umm, no, but you are right, I just do the get_ports by attr and then the trunk list | 15:46 |
| apuimedo | ltomasbo: that is not 2k calls to neutron | 15:46 |
| apuimedo | it shouldn't be | 15:46 |
| ltomasbo | yep, sorry, I'm looking at the code now | 15:46 |
| apuimedo | ok | 15:46 |
| ltomasbo | just 2 neutron calls, the get_ports_by_attrs and the list_trunks | 15:47 |
| ltomasbo | and, one more per trunk | 15:48 |
| ltomasbo | to get the trunk parent port ip | 15:48 |
| ltomasbo | still, it should be faster than that I guess | 15:49 |
| ltomasbo | apuimedo, perhaps you can include a print for each trunk | 15:49 |
| ltomasbo | so that you know if it is stuck or still working | 15:49 |
| apuimedo | ltomasbo: now I don't want to stop it xD | 15:51 |
| ltomasbo | apuimedo, I understand why... | 15:51 |
| apuimedo | but it still didn't give the log message | 15:52 |
| ltomasbo | can you check if the process is actually doing anything? | 15:52 |
| *** yamamoto has joined #openstack-kuryr | 15:52 | |
| apuimedo | ltomasbo: 11-15% cpu | 15:52 |
| ltomasbo | ok, at least doing something | 15:53 |
| apuimedo | ltomasbo: just finished | 15:53 |
| apuimedo | damn! | 15:54 |
| ltomasbo | \o/ | 15:54 |
| apuimedo | It took 14 minutes | 15:54 |
| ltomasbo | how long it took? | 15:54 |
| apuimedo | that's terrible | 15:54 |
| apuimedo | :/ | 15:54 |
| ltomasbo | yep, not good | 15:54 |
| apuimedo | 14min14s | 15:54 |
| apuimedo | I'll optimize this later | 15:54 |
| ltomasbo | should we optimized by reading from a file? or something like that | 15:54 |
| apuimedo | first I'll see how fast we can get it without that | 15:55 |
| ltomasbo | nice! | 15:56 |
| ltomasbo | I'm on the tmux... | 15:56 |
| apuimedo | wow | 15:56 |
| apuimedo | this was fast | 15:56 |
| ltomasbo | that is a good ratio! | 15:56 |
| ltomasbo | yep | 15:56 |
| apuimedo | let's count it | 15:56 |
| ltomasbo | have to be near 50 p/s? | 15:56 |
| apuimedo | weird | 15:57 |
| apuimedo | i'm dumb | 15:57 |
| apuimedo | forgot to pass the params | 15:57 |
| ltomasbo | what happend? | 15:58 |
| apuimedo | xD | 15:58 |
| ltomasbo | ahh, ok jejejej | 15:58 |
| ltomasbo | the namespace... | 15:58 |
| *** yamamoto has quit IRC | 16:03 | |
| apuimedo | good lord | 16:03 |
| apuimedo | that is fast | 16:03 |
| *** egonzalez has quit IRC | 16:03 | |
| ltomasbo | final number? | 16:06 |
| ltomasbo | (or plot) | 16:06 |
| ltomasbo | xD | 16:06 |
| apuimedo | 21.95pods/s | 16:06 |
| apuimedo | so you were right, about 40% faster than not sending the port udpate | 16:07 |
| ltomasbo | :D | 16:07 |
| apuimedo | I honestly expected it to be much faster | 16:07 |
| ltomasbo | would have been better to have the 400% you where thinking about | 16:08 |
| apuimedo | but I guess I can't complain | 16:08 |
| ltomasbo | :d | 16:08 |
| ltomasbo | I guess they are good enough | 16:08 |
| apuimedo | mmm | 16:09 |
| ltomasbo | but still room for improvement! | 16:09 |
| apuimedo | well, the target is about 112pods/s | 16:09 |
| apuimedo | but it depends on the scheduler as well | 16:09 |
| apuimedo | so 1 pod/s/node | 16:09 |
| ltomasbo | yep, with the distributed approach with started discussing the other day, I guess we can beat that | 16:10 |
| apuimedo | yeah, of course | 16:16 |
| apuimedo | ok, going to the doctor with my son | 16:16 |
| apuimedo | tty tomororw | 16:16 |
| apuimedo | *tomorrow | 16:16 |
| ltomasbo | bye | 16:29 |
| dulek | janonymous: Ha, found it! I think config initialization was missing in case of daemonized cni (not the daemon itself). | 16:37 |
| *** pcaruana has quit IRC | 16:38 | |
| *** robust has joined #openstack-kuryr | 16:50 | |
| *** gouthamr has quit IRC | 17:01 | |
| *** gouthamr has joined #openstack-kuryr | 17:02 | |
| *** ltomasbo has quit IRC | 17:06 | |
| dulek | janonymous, apuimedo: Not that daemon works, but at least I can see the communication. :) Will continue tomorrow. | 17:10 |
| *** wangbo has quit IRC | 17:15 | |
| *** ltomasbo has joined #openstack-kuryr | 17:30 | |
| apuimedo | dulek: cool | 18:24 |
| dulek | apuimedo: In case you want to follow this, I'm noting stuff that don't work on the review itself. | 18:34 |
| apuimedo | perfect | 18:34 |
| apuimedo | thanks for that | 18:34 |
| *** jerms has quit IRC | 18:50 | |
| *** jerms has joined #openstack-kuryr | 19:01 | |
| *** neex has joined #openstack-kuryr | 19:16 | |
| *** neex has quit IRC | 19:18 | |
| *** egonzalez has joined #openstack-kuryr | 20:01 | |
| *** gouthamr has quit IRC | 20:03 | |
| *** atoth has quit IRC | 21:22 | |
| *** lakerzhou has quit IRC | 21:27 | |
| *** egonzalez has quit IRC | 21:28 | |
| *** gouthamr has joined #openstack-kuryr | 21:35 | |
| *** dougbtv__ has joined #openstack-kuryr | 21:36 | |
| *** aojea has quit IRC | 22:16 | |
| *** yamamoto_ has joined #openstack-kuryr | 22:19 | |
| *** gouthamr has quit IRC | 22:21 | |
| *** robust has quit IRC | 22:27 | |
| *** openstackgerrit has quit IRC | 22:47 | |
| *** gouthamr has joined #openstack-kuryr | 23:01 | |
| *** gouthamr has quit IRC | 23:40 | |
| *** gouthamr has joined #openstack-kuryr | 23:55 | |
| *** lakerzhou has joined #openstack-kuryr | 23:56 | |
| *** openstackgerrit has joined #openstack-kuryr | 23:57 | |
| openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: [WIP] Supprt searching existing port with macaddress https://review.openstack.org/505443 | 23:57 |
| *** lakerzhou1 has joined #openstack-kuryr | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!