| *** gilfoyle has joined #openstack-kuryr | 00:08 | |
| *** gilfoyle has quit IRC | 00:12 | |
| *** oikiki has quit IRC | 00:15 | |
| *** pmannidi has joined #openstack-kuryr | 00:17 | |
| *** caowei has joined #openstack-kuryr | 00:17 | |
| *** premsankar has quit IRC | 00:22 | |
| *** gilfoyle has joined #openstack-kuryr | 00:29 | |
| *** gilfoyle has quit IRC | 00:33 | |
| *** gilfoyle has joined #openstack-kuryr | 00:44 | |
| *** gilfoyle has quit IRC | 00:50 | |
| *** limao has joined #openstack-kuryr | 00:59 | |
| *** kiennt26 has joined #openstack-kuryr | 01:00 | |
| *** gilfoyle has joined #openstack-kuryr | 01:09 | |
| *** gilfoyle has quit IRC | 01:13 | |
| *** gilfoyle has joined #openstack-kuryr | 01:34 | |
| *** yamamoto has joined #openstack-kuryr | 01:36 | |
| *** gilfoyle has quit IRC | 01:38 | |
| *** yamamoto has quit IRC | 01:44 | |
| *** gilfoyle has joined #openstack-kuryr | 01:46 | |
| *** gilfoyle has quit IRC | 01:51 | |
| *** wangbo_ has joined #openstack-kuryr | 01:57 | |
| *** gilfoyle has joined #openstack-kuryr | 01:59 | |
| *** gilfoyle has quit IRC | 02:03 | |
| *** pmannidi has quit IRC | 02:08 | |
| *** pmannidi has joined #openstack-kuryr | 02:08 | |
| *** gilfoyle has joined #openstack-kuryr | 02:11 | |
| *** pmannidi has quit IRC | 02:13 | |
| *** hongbin has joined #openstack-kuryr | 02:13 | |
| *** gilfoyle has quit IRC | 02:16 | |
| *** hongbin_ has joined #openstack-kuryr | 02:19 | |
| *** hongbin has quit IRC | 02:21 | |
| *** yamamoto has joined #openstack-kuryr | 02:36 | |
| *** gilfoyle has joined #openstack-kuryr | 02:54 | |
| *** limao has quit IRC | 02:55 | |
| *** limao has joined #openstack-kuryr | 02:55 | |
| *** wangbo_ has quit IRC | 02:58 | |
| *** gilfoyle has quit IRC | 02:59 | |
| *** wangbo has joined #openstack-kuryr | 03:00 | |
| *** hongbin_ has quit IRC | 03:28 | |
| *** gilfoyle has joined #openstack-kuryr | 03:49 | |
| *** gilfoyle has quit IRC | 03:53 | |
| *** wangbo has quit IRC | 04:09 | |
| *** kiennt26 has quit IRC | 04:15 | |
| *** pmannidi has joined #openstack-kuryr | 04:39 | |
| *** vikasc has quit IRC | 04:40 | |
| *** gilfoyle has joined #openstack-kuryr | 04:43 | |
| *** aojea has joined #openstack-kuryr | 04:46 | |
| *** gilfoyle has quit IRC | 04:47 | |
| *** aojea has quit IRC | 04:50 | |
| *** caowei has quit IRC | 04:54 | |
| *** wangbo has joined #openstack-kuryr | 04:55 | |
| *** vikasc has joined #openstack-kuryr | 05:03 | |
| *** limao has quit IRC | 05:03 | |
| *** limao has joined #openstack-kuryr | 05:03 | |
| *** vikasc has quit IRC | 05:05 | |
| *** vikasc has joined #openstack-kuryr | 05:06 | |
| *** janki has joined #openstack-kuryr | 05:06 | |
| *** gilfoyle has joined #openstack-kuryr | 05:14 | |
| *** wangbo has quit IRC | 05:18 | |
| *** gilfoyle has quit IRC | 05:19 | |
| *** caowei has joined #openstack-kuryr | 05:22 | |
| *** wangbo has joined #openstack-kuryr | 05:22 | |
| *** pcaruana has joined #openstack-kuryr | 05:24 | |
| *** aojea has joined #openstack-kuryr | 05:26 | |
| *** atoth has quit IRC | 05:26 | |
| *** atoth has joined #openstack-kuryr | 05:26 | |
| *** pcaruana has quit IRC | 05:29 | |
| *** oikiki has joined #openstack-kuryr | 05:49 | |
| *** yamamoto_ has joined #openstack-kuryr | 05:53 | |
| *** limao has quit IRC | 05:55 | |
| *** yamamoto has quit IRC | 05:57 | |
| *** limao has joined #openstack-kuryr | 05:58 | |
| *** aojea has quit IRC | 06:00 | |
| *** wangbo has quit IRC | 06:04 | |
| *** gilfoyle has joined #openstack-kuryr | 06:08 | |
| *** aojea has joined #openstack-kuryr | 06:10 | |
| *** gilfoyle has quit IRC | 06:13 | |
| *** kiennt26 has joined #openstack-kuryr | 06:16 | |
| *** livelace-link has joined #openstack-kuryr | 06:24 | |
| livelace | test | 06:24 |
|---|---|---|
| livelace-link | test back | 06:24 |
| *** livelace-link has quit IRC | 06:26 | |
| *** livelace-link has joined #openstack-kuryr | 06:27 | |
| *** livelace-link has quit IRC | 06:36 | |
| *** livelace-link has joined #openstack-kuryr | 06:36 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/kuryr-kubernetes master: [WIP] CNI Unit Tests Framework https://review.openstack.org/508106 | 06:43 |
| *** livelace-link has quit IRC | 06:47 | |
| *** livelace-link has joined #openstack-kuryr | 06:58 | |
| *** pcaruana has joined #openstack-kuryr | 07:04 | |
| *** pmannidi has quit IRC | 07:07 | |
| *** garyloug has joined #openstack-kuryr | 07:15 | |
| *** kiennt26 has quit IRC | 07:20 | |
| ltomasbo | apuimedo, I tried your script to create the port, and I'm still seeing the bug: https://bugs.launchpad.net/neutron/+bug/1696051 | 07:25 |
| openstack | Launchpad bug 1696051 in neutron "Missing binding details for bulk port creation requests" [Undecided,New] | 07:25 |
| ltomasbo | apuimedo, http://paste.openstack.org/show/622281/ | 07:25 |
| ltomasbo | even though if I later do a port show, those details are there: | 07:25 |
| ltomasbo | | binding_vif_details | datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | 07:25 |
| ltomasbo | | binding_vif_type | ovs | 07:25 |
| ltomasbo | apuimedo, good thing is that those ports are being recovered upon controller reboot | 07:28 |
| *** aojea has quit IRC | 07:30 | |
| *** egonzalez has joined #openstack-kuryr | 07:44 | |
| apuimedo | ltomasbo: ah! You meant about the creation answer? | 07:53 |
| apuimedo | Well, it doesn't matter, immediately after they have the details | 07:54 |
| ltomasbo | apuimedo, yes | 07:54 |
| apuimedo | don't they? | 07:54 |
| ltomasbo | sure, but you need to do another get then | 07:54 |
| ltomasbo | apuimedo, it just annoyed me that if you do a normal creation of a port, it has that information | 07:55 |
| ltomasbo | but if you do a bulk creation of a single port, then it is not there | 07:55 |
| *** gilfoyle has joined #openstack-kuryr | 07:56 | |
| ltomasbo | and we had to add a workaround due to that: https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/neutron_vif.py#L59-L66 | 07:56 |
| apuimedo | ltomasbo: about that workaround... Probably a second get is faster than the single port creation | 07:58 |
| ltomasbo | yes, but not sure if we will always have that information in the second get or we may end up in a race | 07:59 |
| ltomasbo | anyhow, I can do a while with sleep 1 or so, with a few tries | 07:59 |
| apuimedo | ltomasbo: mmmm | 07:59 |
| apuimedo | I think it's better to check if there is a race or not | 07:59 |
| apuimedo | ltomasbo: however, what worries me is the cni timeouts I was seeing yesterday on baremetal that make PLEG put the node as not ready | 08:00 |
| apuimedo | since livelace-link also reported it | 08:00 |
| *** gilfoyle has quit IRC | 08:01 | |
| ltomasbo | yes, that is a hard problem | 08:01 |
| ltomasbo | apuimedo, do you know if it is only openshift, or does it happen to with k8s? | 08:01 |
| livelace | Hello guys, I'm going to understand the PLEG problem at the weekend too | 08:03 |
| livelace | Last time, when I enabled all debugs, the PLEG problem was gone and I couldn't reproduce it, it happens at the beginning of my deployments | 08:06 |
| apuimedo | livelace: I'm struggling with it in a 26 node deployemnt | 08:07 |
| apuimedo | *deployment | 08:07 |
| apuimedo | 2-3 nodes start going notready then ready and others go notready | 08:07 |
| livelace | After some time/some special fluctuations they may gone, but may stay | 08:07 |
| *** oikiki has quit IRC | 08:07 | |
| apuimedo | livelace: did you see cni timeouts in your kubelet logs? | 08:07 |
| livelace | apuimedo, I'm not sure that it is cni timeouts, because PLEG appears without cni timeouts too | 08:08 |
| livelace | I guess it happens during interface binding to the Neutron ports | 08:09 |
| apuimedo | livelace: I always got the impression that docker runtime PLEG, once it sees some issues loses all trustworthiness | 08:09 |
| apuimedo | xD | 08:09 |
| livelace | During this time | 08:09 |
| livelace | Folks from https://github.com/kubernetes/kubernetes/issues/45419 met the PLEG problem with another network solutions (weave,calico) too | 08:10 |
| livelace | I think kubelet/origin-node donesn't like network switching timeouts, or other such things | 08:11 |
| *** limao has quit IRC | 08:12 | |
| *** limao has joined #openstack-kuryr | 08:12 | |
| apuimedo | livelace: I agree | 08:13 |
| ltomasbo | so, do you have in mind any workaround until this is discovered/fixed? | 08:17 |
| *** limao has quit IRC | 08:17 | |
| *** gilfoyle has joined #openstack-kuryr | 08:20 | |
| *** oikiki has joined #openstack-kuryr | 08:20 | |
| *** livelace-link has quit IRC | 08:22 | |
| *** livelace-link has joined #openstack-kuryr | 08:22 | |
| dulek | irenab: Hi, can you take a look on https://review.openstack.org/#/c/505125/ ? We need it merged to upload fixed Docker images to DockerHub. | 08:24 |
| livelace | ltomasbo, If question was for me - I don't know, as I told - PLEG problem may be gone, need to find patterns of this behavior | 08:24 |
| *** alishaaneja has quit IRC | 08:27 | |
| *** limao has joined #openstack-kuryr | 08:30 | |
| janonymous | dmellado: I guess Zuul change gate failure: http://logs.openstack.org/06/508106/2/check/openstack-tox-py35/24bc008/job-output.txt.gz , will your patch to add .zuul will resolve this? | 08:34 |
| dmellado | janonymous: lemme see | 08:35 |
| dmellado | awesome, I knew this was going to be broken | 08:36 |
| dmellado | janonymous: could you paste your patch? | 08:37 |
| apuimedo | ltomasbo: workaround for what? | 08:44 |
| apuimedo | dulek: irenab is on weekend (Israel weekdays) | 08:44 |
| openstackgerrit | Daniel Mellado proposed openstack/kuryr-kubernetes master: DNM Test for Zuulv3 https://review.openstack.org/508440 | 08:45 |
| dulek | apuimedo: Oh, okay. | 08:45 |
| dmellado | morning folks | 08:45 |
| dmellado | as I expected, it seems the migration to zuulv3 disn't go well | 08:45 |
| dulek | dmellado: As everyone expected? ;) | 08:45 |
| dmellado | let's see what that patch has to say after janonymous' trial patch xD | 08:46 |
| dmellado | dulek: totally xD | 08:46 |
| apuimedo | dmellado: it went well. I get very suspicious if upgrades don't break | 08:47 |
| *** wangbo has joined #openstack-kuryr | 08:47 | |
| apuimedo | it means something is deeply wrong with the matrix | 08:47 |
| dmellado | apuimedo: well, I might be having a dejà-vu interview on monday (which I'll just forward to you, of course) xD | 08:48 |
| *** wangbo has quit IRC | 08:49 | |
| apuimedo | dmellado: but you'll be there, right? | 08:49 |
| *** wangbo has joined #openstack-kuryr | 08:49 | |
| dmellado | well, we can flip a coin | 08:49 |
| dmellado | xD | 08:49 |
| dmellado | nah, I'll be around too | 08:49 |
| *** danil has joined #openstack-kuryr | 08:50 | |
| apuimedo | dmellado: lion I go number I don't (you go in both cases) https://www.random.org/coins/?num=1&cur=60-czk.20kr | 08:51 |
| apuimedo | and don't cheat | 08:51 |
| dmellado | loool | 08:51 |
| dmellado | I don't accept those duel rules | 08:52 |
| dmellado | xD | 08:52 |
| janonymous | dmellado: Do i have to update anything :P | 08:54 |
| dmellado | dulek: well, the good thing is that after fixing this we'd be able to properly deploy some new multinode gates | 08:54 |
| dmellado | janonymous: just hold in there, I wanted to add a 'foo' patch | 08:54 |
| dmellado | after it, I'll go nagging to the infra folks | 08:54 |
| *** livelace has quit IRC | 08:54 | |
| janonymous | dmellado: :D | 08:55 |
| *** livelace has joined #openstack-kuryr | 09:00 | |
| *** livelace has quit IRC | 09:01 | |
| *** yamamoto_ has quit IRC | 09:01 | |
| *** oikiki has quit IRC | 09:04 | |
| apuimedo | ltomasbo: I see you are using my 'kaboom' naming for ports in pools | 09:10 |
| apuimedo | dmellado: we really, really need multinode gates | 09:11 |
| ltomasbo | apuimedo, yep! :D | 09:11 |
| dmellado | apuimedo: you tell me | 09:11 |
| dmellado | xD | 09:11 |
| dmellado | ltomasbo: be aware | 09:11 |
| apuimedo | I want one with dns | 09:11 |
| dmellado | kabooom comes from something with 3 o's | 09:12 |
| dmellado | xD | 09:12 |
| apuimedo | dmellado: xD | 09:12 |
| ltomasbo | xD | 09:12 |
| *** aojea has joined #openstack-kuryr | 09:15 | |
| *** gilfoyle_ has joined #openstack-kuryr | 09:25 | |
| *** gilfoyle has quit IRC | 09:27 | |
| *** yamamoto has joined #openstack-kuryr | 09:29 | |
| *** yamamoto has quit IRC | 09:32 | |
| dmellado | janonymous: I guess you might need to just recheck | 09:33 |
| dmellado | check http://zuulv3.openstack.org/ | 09:33 |
| dmellado | for kuryr-kubernetes | 09:33 |
| dmellado | basically It doesn't seem that the state is stable at all yet | 09:33 |
| janonymous | dmellado: ohh, i see ..i wil wait , it's wip patch though :D | 09:34 |
| dmellado | if you check my patch there, which just adds a 'foo' it seems that it's going so far | 09:34 |
| dmellado | will wait for it to end | 09:34 |
| dmellado | but word of warning | 09:34 |
| dmellado | expect breakages for the time being, as I'll be modifying the current gates heavily | 09:35 |
| dmellado | ping me for any issues ;) | 09:35 |
| janonymous | dmellado: thanks! :) | 09:35 |
| *** yamamoto has joined #openstack-kuryr | 09:40 | |
| *** aojea has quit IRC | 09:49 | |
| *** yamamoto has quit IRC | 09:54 | |
| *** wangbo has quit IRC | 09:56 | |
| *** yamamoto has joined #openstack-kuryr | 10:01 | |
| *** egonzalez has quit IRC | 10:05 | |
| *** yuval has quit IRC | 10:06 | |
| *** jgriffith has quit IRC | 10:08 | |
| *** caowei has quit IRC | 10:08 | |
| *** yuval has joined #openstack-kuryr | 10:09 | |
| *** aojea has joined #openstack-kuryr | 10:09 | |
| *** aojea has quit IRC | 10:12 | |
| *** aojea has joined #openstack-kuryr | 10:12 | |
| *** jgriffith has joined #openstack-kuryr | 10:14 | |
| *** aojea_ has joined #openstack-kuryr | 10:18 | |
| *** aojea has quit IRC | 10:19 | |
| *** egonzalez has joined #openstack-kuryr | 10:20 | |
| *** yedongcan has joined #openstack-kuryr | 10:28 | |
| *** aojea_ has quit IRC | 10:35 | |
| *** yedongcan has left #openstack-kuryr | 10:47 | |
| *** aojea has joined #openstack-kuryr | 10:50 | |
| openstackgerrit | Antoni Segura Puimedon proposed openstack/kuryr-kubernetes master: k8s_client: use requests session pooling https://review.openstack.org/508488 | 11:06 |
| *** wangbo has joined #openstack-kuryr | 11:13 | |
| *** vikasc has quit IRC | 11:20 | |
| *** aojea has quit IRC | 11:27 | |
| *** vikasc has joined #openstack-kuryr | 11:33 | |
| *** alishaaneja has joined #openstack-kuryr | 11:34 | |
| *** garyloug has quit IRC | 11:43 | |
| *** garyloug has joined #openstack-kuryr | 11:43 | |
| *** wangbo has quit IRC | 11:59 | |
| *** vikasc has quit IRC | 12:17 | |
| *** aojea has joined #openstack-kuryr | 12:20 | |
| *** vikasc has joined #openstack-kuryr | 12:37 | |
| *** kiennt26 has joined #openstack-kuryr | 12:43 | |
| *** Rince_ has quit IRC | 12:47 | |
| *** danil has quit IRC | 12:50 | |
| *** kiennt26 has quit IRC | 12:58 | |
| *** kiennt26 has joined #openstack-kuryr | 12:59 | |
| *** garyloug has quit IRC | 13:05 | |
| *** kiennt26 has quit IRC | 13:08 | |
| *** gouthamr has joined #openstack-kuryr | 13:44 | |
| *** hongbin has joined #openstack-kuryr | 13:57 | |
| *** limao has quit IRC | 13:58 | |
| *** limao has joined #openstack-kuryr | 13:59 | |
| *** kiennt26 has joined #openstack-kuryr | 14:00 | |
| *** limao has quit IRC | 14:03 | |
| *** aojea has quit IRC | 14:16 | |
| *** aojea has joined #openstack-kuryr | 14:24 | |
| *** vikasc has quit IRC | 14:37 | |
| *** aojea has quit IRC | 14:42 | |
| *** apuimedo has quit IRC | 14:43 | |
| *** aojea has joined #openstack-kuryr | 14:46 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Recover precreated ports at NeutronVIFPool driver https://review.openstack.org/507932 | 14:49 |
| *** egonzalez has quit IRC | 15:02 | |
| *** aojea has quit IRC | 15:05 | |
| *** apuimedo has joined #openstack-kuryr | 15:10 | |
| *** aojea has joined #openstack-kuryr | 15:15 | |
| *** yamamoto has quit IRC | 15:18 | |
| *** aojea has quit IRC | 15:25 | |
| *** kiennt26 has quit IRC | 15:30 | |
| *** alishaaneja has quit IRC | 15:39 | |
| *** janki has quit IRC | 15:43 | |
| *** yamamoto has joined #openstack-kuryr | 16:18 | |
| *** yamamoto has quit IRC | 16:24 | |
| *** pcaruana has quit IRC | 16:26 | |
| *** gouthamr has quit IRC | 17:10 | |
| *** yamamoto has joined #openstack-kuryr | 17:20 | |
| *** yamamoto has quit IRC | 17:25 | |
| *** gouthamr has joined #openstack-kuryr | 17:33 | |
| *** gianpietro has joined #openstack-kuryr | 17:51 | |
| *** oikiki has joined #openstack-kuryr | 18:16 | |
| *** yamamoto has joined #openstack-kuryr | 18:22 | |
| *** yamamoto has quit IRC | 18:26 | |
| *** vikasc has joined #openstack-kuryr | 19:06 | |
| *** gilfoyle_ has quit IRC | 19:11 | |
| *** yamamoto has joined #openstack-kuryr | 19:23 | |
| *** yamamoto has quit IRC | 19:28 | |
| *** gilfoyle has joined #openstack-kuryr | 19:51 | |
| *** gilfoyle has quit IRC | 19:56 | |
| *** gilfoyle has joined #openstack-kuryr | 20:04 | |
| *** gilfoyle has quit IRC | 20:08 | |
| *** aojea has joined #openstack-kuryr | 20:09 | |
| *** aojea has quit IRC | 20:13 | |
| *** yamamoto has joined #openstack-kuryr | 20:24 | |
| *** yamamoto has quit IRC | 20:29 | |
| *** gilfoyle has joined #openstack-kuryr | 20:40 | |
| *** gilfoyle has quit IRC | 20:44 | |
| *** gilfoyle has joined #openstack-kuryr | 20:57 | |
| *** aojea has joined #openstack-kuryr | 21:18 | |
| *** gilfoyle has quit IRC | 21:19 | |
| *** ltomasbo has quit IRC | 21:24 | |
| *** yamamoto has joined #openstack-kuryr | 21:25 | |
| *** ltomasbo has joined #openstack-kuryr | 21:28 | |
| *** yamamoto has quit IRC | 21:31 | |
| *** oikiki has quit IRC | 22:19 | |
| *** yamamoto has joined #openstack-kuryr | 22:27 | |
| *** gouthamr has quit IRC | 22:29 | |
| *** aojea has quit IRC | 22:32 | |
| *** yamamoto has quit IRC | 22:32 | |
| openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: [WIP] Supprt searching existing port with macaddress https://review.openstack.org/505443 | 22:34 |
| *** aojea has joined #openstack-kuryr | 22:40 | |
| *** aojea has quit IRC | 22:45 | |
| *** aojea has joined #openstack-kuryr | 22:57 | |
| *** aojea has quit IRC | 23:02 | |
| *** hongbin has quit IRC | 23:09 | |
| *** yamamoto has joined #openstack-kuryr | 23:29 | |
| *** yamamoto has quit IRC | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!