| *** tonanhngo has joined #openstack-kuryr | 00:08 | |
| *** mattmceuen has quit IRC | 00:09 | |
| dougbtv | hrmmm, it complained that another etcd container was running under the same name | 00:11 |
|---|---|---|
| dougbtv | I ran ./unstack.sh, but, I'm wondering if it didn't clean up an already running container | 00:11 |
| dougbtv | to be sure, I'm gonna wipe that machine and rebuild it (it's trivial, it's provisioned with bifrost/ironic) | 00:11 |
| *** tonanhngo has quit IRC | 00:13 | |
| apuimedo | let me see | 00:15 |
| apuimedo | it probably failed to delete etcd on unstack | 00:16 |
| apuimedo | I think we may have removed that, but it should not have tried to start it again then | 00:16 |
| apuimedo | I'll make a note to check it | 00:16 |
| * apuimedo heading to bed | 00:16 | |
| dougbtv | thanks man, no big worries but figured I'd say something | 00:16 |
| dougbtv | gnite!! | 00:16 |
| apuimedo | dougbtv: thanks for going through the deployment so many times ;-) | 00:17 |
| dougbtv | hey no prob :) I'm psyched that you got a fix in and for the help, happy to do that at least for sure | 00:17 |
| *** limao has joined #openstack-kuryr | 00:23 | |
| *** limao has quit IRC | 00:24 | |
| *** limao has joined #openstack-kuryr | 00:25 | |
| *** salv-orlando has quit IRC | 00:31 | |
| *** hongbin has quit IRC | 00:39 | |
| *** saneax is now known as saneax-_-|AFK | 00:40 | |
| *** tonanhngo has joined #openstack-kuryr | 00:45 | |
| *** tonanhngo has quit IRC | 00:50 | |
| *** salv-orlando has joined #openstack-kuryr | 01:01 | |
| *** salv-orlando has quit IRC | 01:06 | |
| *** portdirect is now known as sdake______ | 01:09 | |
| *** sdake______ is now known as portdirect | 01:09 | |
| *** dougbtv has quit IRC | 01:21 | |
| *** tonanhngo has joined #openstack-kuryr | 01:49 | |
| *** tonanhngo has quit IRC | 01:53 | |
| *** salv-orlando has joined #openstack-kuryr | 02:02 | |
| *** salv-orlando has quit IRC | 02:07 | |
| *** tonanhngo has joined #openstack-kuryr | 02:19 | |
| *** tianquan has joined #openstack-kuryr | 02:21 | |
| *** tonanhngo has quit IRC | 02:24 | |
| *** lihi has quit IRC | 02:36 | |
| *** irenab_ has quit IRC | 02:36 | |
| *** yuvalb has quit IRC | 02:37 | |
| *** irenab_ has joined #openstack-kuryr | 02:37 | |
| *** yuvalb has joined #openstack-kuryr | 02:38 | |
| *** lihi has joined #openstack-kuryr | 02:39 | |
| *** hongbin has joined #openstack-kuryr | 03:01 | |
| *** salv-orlando has joined #openstack-kuryr | 03:03 | |
| *** salv-orlando has quit IRC | 03:08 | |
| *** hongbin has quit IRC | 03:24 | |
| *** hongbin has joined #openstack-kuryr | 03:25 | |
| *** hongbin_ has joined #openstack-kuryr | 03:42 | |
| *** hongbin has quit IRC | 03:44 | |
| *** hongbin has joined #openstack-kuryr | 03:54 | |
| *** hongbin_ has quit IRC | 03:55 | |
| *** salv-orlando has joined #openstack-kuryr | 04:04 | |
| *** salv-orlando has quit IRC | 04:09 | |
| *** limao has quit IRC | 04:11 | |
| *** hongbin_ has joined #openstack-kuryr | 04:12 | |
| *** limao has joined #openstack-kuryr | 04:12 | |
| *** hongbin has quit IRC | 04:16 | |
| *** limao has quit IRC | 04:17 | |
| *** hongbin_ has quit IRC | 04:30 | |
| *** tianquan has quit IRC | 04:31 | |
| *** limao has joined #openstack-kuryr | 04:57 | |
| *** salv-orlando has joined #openstack-kuryr | 05:05 | |
| *** salv-orlando has quit IRC | 05:09 | |
| *** tianquan has joined #openstack-kuryr | 05:24 | |
| openstackgerrit | vikas choudhary proposed openstack/kuryr-kubernetes master: Add support for HTTPS client https://review.openstack.org/440979 | 05:31 |
| vikasc | irenab, ping | 05:32 |
| *** yamamoto_ has joined #openstack-kuryr | 06:01 | |
| *** salv-orlando has joined #openstack-kuryr | 06:06 | |
| *** janki has joined #openstack-kuryr | 06:09 | |
| *** salv-orlando has quit IRC | 06:11 | |
| *** limao has quit IRC | 06:13 | |
| *** limao_ has joined #openstack-kuryr | 06:13 | |
| *** yedongcan has joined #openstack-kuryr | 06:30 | |
| *** yamamoto_ has quit IRC | 06:36 | |
| *** salv-orlando has joined #openstack-kuryr | 06:37 | |
| *** pcaruana has joined #openstack-kuryr | 06:43 | |
| *** jchhatbar has joined #openstack-kuryr | 06:50 | |
| *** janki has quit IRC | 06:53 | |
| *** saneax-_-|AFK is now known as saneax | 07:05 | |
| *** salv-orl_ has joined #openstack-kuryr | 07:23 | |
| *** yamamoto_ has joined #openstack-kuryr | 07:23 | |
| *** salv-orlando has quit IRC | 07:26 | |
| *** ltomasbo|away is now known as ltomasbo | 07:32 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Generic vif driver extension to enable ports reuse https://review.openstack.org/436876 | 07:41 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Generic vif driver extension to enable ports reuse https://review.openstack.org/436876 | 07:47 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Generic vif pool driver extension to precreate reusable ports https://review.openstack.org/436877 | 07:49 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Nested vif driver extension to enable ports reuse https://review.openstack.org/436893 | 07:51 |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Nested vlan vif pool driver extension to precreate reusable subports https://review.openstack.org/436894 | 07:52 |
| *** salv-orl_ has quit IRC | 08:00 | |
| *** jchhatbar is now known as janki | 08:12 | |
| apuimedo | ivc_: vikasc: irenab: ltomasbo: pike managed projects feature freeze for pike is July 24th | 08:24 |
| dmellado | morning guys | 08:26 |
| dmellado | apuimedo: so you were packaging with princess mononoke sountrack? lol | 08:27 |
| dmellado | deb or rpm? xD | 08:27 |
| apuimedo | dmellado: rpm | 08:27 |
| dmellado | apuimedo: so then it was 'spec'tacular | 08:27 |
| * dmellado goes into hiding | 08:27 | |
| apuimedo | for deb I'll probably need to resort to "Songs of Love and Hate" by Leonard Cohen | 08:27 |
| apuimedo | dmellado: do not hide... You'll be the one reviewing it! | 08:28 |
| apuimedo | xd | 08:28 |
| dmellado | :O | 08:28 |
| dmellado | xD | 08:28 |
| dmellado | btw, how's bcn? here I'm having a terrible mix in between cold and allergy | 08:28 |
| dmellado | is there pollen around there yet? | 08:28 |
| apuimedo | dmellado: I don't feel it in the town | 08:29 |
| apuimedo | in Mordor (Barcelona) there'll probably be pollution high enough to trigger my allergy | 08:29 |
| apuimedo | I don't know how people can prefer to live in such polluted cities (those that have a choice, obviously) | 08:30 |
| ltomasbo | :D | 08:32 |
| apuimedo | ltomasbo: dmellado: http://acephalous.typepad.com/.a/6a00d8341c2df453ef013488046dc0970c-500wi | 08:33 |
| apuimedo | that's me going to Barcelona with my donkey | 08:33 |
| dmellado | apuimedo: in Moriah (Madrid) seems like it's there the same thing | 08:33 |
| ltomasbo | lol | 08:33 |
| dmellado | heh xD | 08:33 |
| dmellado | ltomasbo: please keep going with my work if I die xD | 08:33 |
| dmellado | I'm feeling really terrible today xD | 08:34 |
| ltomasbo | dmellado, I already have enough with mine! :D | 08:34 |
| dmellado | xD | 08:34 |
| apuimedo | dmellado: https://www.youtube.com/watch?v=r622O7Kh4B4 if you die, I have ways to bring you back and help with testing and packaging | 08:36 |
| dmellado | LOL | 08:36 |
| dmellado | xD | 08:36 |
| dmellado | apuimedo: take into consideration that that might atract gabert! errr gollum xD | 08:37 |
| apuimedo | xD | 08:37 |
| ltomasbo | apuimedo, vikasc, irenab, ivc_: I updated the ports pool patch sets with the vif/pool driver split and the ResourceNotReady | 08:45 |
| ltomasbo | so you can take a look when you want/can | 08:45 |
| ltomasbo | I will update the devref/blueprint later today | 08:45 |
| vikasc | ltomasbo, sure, thanks! | 08:45 |
| *** reedip_ has quit IRC | 08:50 | |
| *** pmannidi has quit IRC | 08:51 | |
| *** salv-orlando has joined #openstack-kuryr | 09:00 | |
| *** reedip_ has joined #openstack-kuryr | 09:02 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** salv-orlando has quit IRC | 09:05 | |
| apuimedo | great! | 09:05 |
| *** limao_ has quit IRC | 09:07 | |
| *** reedip_ has quit IRC | 09:18 | |
| *** salv-orlando has joined #openstack-kuryr | 09:31 | |
| *** reedip_ has joined #openstack-kuryr | 09:31 | |
| *** salv-orlando has quit IRC | 09:35 | |
| *** yamamoto_ has quit IRC | 09:35 | |
| *** garyloug has joined #openstack-kuryr | 09:37 | |
| ltomasbo | apuimedo, vikasc, irenab: also take a look at the blueprint for approval/modification https://blueprints.launchpad.net/kuryr-kubernetes/+spec/ports-pool | 09:38 |
| apuimedo | will do | 09:39 |
| ltomasbo | thanks! | 09:40 |
| vikasc | apuimedo, thanks ltomasbo , will try to go through asap! | 09:42 |
| apuimedo | ltomasbo: vikasc: irenab: I went throught the blueprint and marked it for pike-2 | 09:47 |
| apuimedo | hopefully we can have it before that | 09:48 |
| apuimedo | June 10th | 09:48 |
| apuimedo | *June 5th | 09:48 |
| ltomasbo | apuimedo, I hope so too! | 09:48 |
| ltomasbo | I'm testing the new patch series | 09:48 |
| ltomasbo | and then I will work on the rebooting part (discovering kuryr ports/subports) | 09:49 |
| apuimedo | ltomasbo: the port filtering blueprint? | 09:49 |
| *** tianquan has quit IRC | 09:50 | |
| ltomasbo | I guess is somehow related to that | 09:50 |
| ltomasbo | but we needed for the ports pool | 09:50 |
| ltomasbo | if we create X ports for the pools | 09:50 |
| ltomasbo | and then reboot the kuryr-controller | 09:50 |
| apuimedo | https://trello.com/c/nigsLmOS/64-port-filtering-blueprint | 09:50 |
| ltomasbo | we need to discover the ports created (and not used) to include them again into the pools | 09:50 |
| apuimedo | yes, it is related | 09:51 |
| ltomasbo | yep, that one | 09:51 |
| apuimedo | ltomasbo: you are on the hook https://launchpad.net/kuryr-kubernetes/+milestone/pike-2 | 10:00 |
| apuimedo | :P | 10:00 |
| ltomasbo | :D | 10:01 |
| *** ltomasbo is now known as ltomasbo|away | 10:14 | |
| *** salv-orlando has joined #openstack-kuryr | 10:20 | |
| *** yamamoto has joined #openstack-kuryr | 10:45 | |
| *** yamamoto has quit IRC | 10:48 | |
| *** yamamoto has joined #openstack-kuryr | 10:50 | |
| *** tianquan has joined #openstack-kuryr | 10:50 | |
| *** tianquan has quit IRC | 10:50 | |
| *** tianquan has joined #openstack-kuryr | 10:50 | |
| *** yamamoto has quit IRC | 10:51 | |
| *** dimak_ has joined #openstack-kuryr | 10:52 | |
| *** reedip_ has quit IRC | 10:53 | |
| *** ltomasbo|away is now known as ltomasbo | 10:55 | |
| *** tianquan has quit IRC | 11:01 | |
| *** openstackgerrit has joined #openstack-kuryr | 11:04 | |
| openstackgerrit | Dongcan Ye proposed openstack/kuryr-libnetwork master: Add support kuryr IPv6 subnet https://review.openstack.org/442294 | 11:04 |
| *** reedip_ has joined #openstack-kuryr | 11:07 | |
| *** yedongcan has left #openstack-kuryr | 11:14 | |
| *** yamamoto has joined #openstack-kuryr | 11:15 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** yamamoto has joined #openstack-kuryr | 11:18 | |
| *** yamamoto has quit IRC | 11:19 | |
| *** tianquan has joined #openstack-kuryr | 11:23 | |
| *** reedip_ has quit IRC | 11:29 | |
| *** reedip_ has joined #openstack-kuryr | 11:42 | |
| openstackgerrit | Merged openstack/kuryr master: Add unit tests for nested binding driver https://review.openstack.org/441743 | 11:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/438604 | 12:24 |
| *** david-lyle has quit IRC | 12:30 | |
| *** david-lyle has joined #openstack-kuryr | 12:33 | |
| *** salv-orlando has quit IRC | 12:48 | |
| *** garyloug_ has joined #openstack-kuryr | 12:50 | |
| *** garyloug has quit IRC | 12:52 | |
| *** salv-orlando has joined #openstack-kuryr | 12:54 | |
| *** mattmceuen has joined #openstack-kuryr | 13:10 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Kuryr Kubernetes Resource Manager design reference document https://review.openstack.org/427681 | 13:11 |
| *** dougbtv has joined #openstack-kuryr | 13:16 | |
| *** salv-orl_ has joined #openstack-kuryr | 13:23 | |
| *** salv-orlando has quit IRC | 13:26 | |
| *** tianquan has quit IRC | 13:40 | |
| mchiappero | # TODO(ivc): tune retry interval/timeout | 14:08 |
| mchiappero | I'm seeing plenty of ResourceNotReady exceptions | 14:08 |
| mchiappero | any suggestion? | 14:09 |
| apuimedo | mchiappero: which part? | 14:11 |
| mchiappero | the handler | 14:11 |
| apuimedo | for pods? | 14:11 |
| apuimedo | vif handler | 14:11 |
| mchiappero | yes | 14:11 |
| apuimedo | ? | 14:11 |
| apuimedo | mchiappero: on waiting for the port becoming active? | 14:12 |
| mchiappero | I don't know anymore actually | 14:12 |
| mchiappero | I started some test, came back | 14:12 |
| mchiappero | and noticed that only when deleting | 14:12 |
| apuimedo | vikasc: irenab: https://review.openstack.org/#/c/442867/ | 14:12 |
| apuimedo | mchiappero: so it happens mostly when deleting? | 14:13 |
| mchiappero | I guess no because the test created an increasing number of pods, and one failed with a timeout | 14:13 |
| mchiappero | then I suppos kuryr entered in an invalid state | 14:13 |
| apuimedo | mchiappero: with replication controller? | 14:15 |
| mchiappero | no | 14:15 |
| mchiappero | 1 replica | 14:15 |
| apuimedo | meeting | 14:16 |
| apuimedo | back in 15m | 14:16 |
| mchiappero | np | 14:16 |
| mchiappero | uhm yes, definitely deleting as well | 14:17 |
| *** salv-orl_ has quit IRC | 14:18 | |
| *** mattmceuen has quit IRC | 14:20 | |
| mchiappero | Failed to delete port with name or ID 'fe773864-3a6c-4a6a-84aa-405e8faf1a33': HttpException: Conflict, Port fe773864-3a6c-4a6a-84aa-405e8faf1a33 is currently a subport for trunk 318a4cf1-d1b2-4a48-a3d5-47c01af5303b. | 14:21 |
| mchiappero | 1 of 1 ports failed to delete | 14:21 |
| mchiappero | WOW... | 14:21 |
| apuimedo | mchiappero: you are trying ltomasbo's vlan subports? | 14:23 |
| mchiappero | yes | 14:23 |
| mchiappero | but it's actually neutron although I restarted all q- services | 14:23 |
| ltomasbo | mchiappero, which patches? | 14:23 |
| ltomasbo | the ones for the pool of ports? | 14:24 |
| mchiappero | no, master | 14:24 |
| apuimedo | mchiappero: well, somehow there was not removal from subport before deleting the port, I suppose | 14:24 |
| ltomasbo | and for the nested or the baremetal case? | 14:24 |
| mchiappero | nested | 14:24 |
| ltomasbo | but master branch? | 14:24 |
| mchiappero | apuimedo: what do you mean? | 14:24 |
| mchiappero | master | 14:24 |
| ltomasbo | then is vikasc patches? | 14:24 |
| mchiappero | no | 14:24 |
| mchiappero | I haven't checked lately but what has been merged weeks ago | 14:25 |
| ltomasbo | ok ok | 14:25 |
| *** tianquan has joined #openstack-kuryr | 14:25 | |
| mchiappero | ok, let me check this stack :) | 14:25 |
| ltomasbo | and for kuberentes? or libnetwork? | 14:25 |
| mchiappero | kubernetes | 14:25 |
| ltomasbo | I got some problems with the nested environment too due to timeouts | 14:26 |
| ltomasbo | and the error you get for the delte port | 14:26 |
| ltomasbo | is because you first need to detach it from the trunk | 14:26 |
| ltomasbo | openstack network trunk unset --subport port_name trunk_name | 14:26 |
| ltomasbo | and then you will be able to delete it | 14:26 |
| mchiappero | but shouldn't kuryr already do that? | 14:26 |
| ltomasbo | yes, it does | 14:27 |
| mchiappero | is it the problem we discussed in a IRC meeting some time ago? | 14:27 |
| ltomasbo | but maybe something broke in between | 14:27 |
| mchiappero | sometimes it works fine, but this time failed | 14:27 |
| ltomasbo | it happened to me when trying nested container in a VM running in another VM with devstack | 14:27 |
| ltomasbo | due to timeouts, and the retry mechanisms | 14:28 |
| ltomasbo | it will work with the new pool port optimization :D | 14:28 |
| apuimedo | mchiappero: ltomasbo: sounds like a bug to report | 14:28 |
| ltomasbo | just kidding... | 14:28 |
| mchiappero | is there a way to mitigate? | 14:28 |
| ltomasbo | not sure that is a bug | 14:28 |
| ltomasbo | I think it is due to lack of resources | 14:28 |
| mchiappero | the comment about tuning, which parameters is it referring to? | 14:28 |
| ltomasbo | it happen to me due to needed a few minutes just to boot a container... | 14:29 |
| mchiappero | like? | 14:29 |
| apuimedo | ltomasbo: even with lack of resources, it sohuld be retrying to unset the subport | 14:29 |
| apuimedo | instead of going directly to try to delete the subport | 14:29 |
| apuimedo | so that's a bug | 14:29 |
| ltomasbo | ahh, when deleting | 14:29 |
| mchiappero | yes | 14:29 |
| mchiappero | well, I would assume both | 14:29 |
| ltomasbo | I'm assuming it will not even go to running state, right? | 14:29 |
| *** yedongcan has joined #openstack-kuryr | 14:29 | |
| mchiappero | resources like? | 14:29 |
| mchiappero | exactly | 14:29 |
| mchiappero | one of 16 failed | 14:30 |
| ltomasbo | maybe I should read what you were discussing before I came back from lunch... | 14:30 |
| mchiappero | uhm dunno | 14:30 |
| mchiappero | so, it's actually not neutron? | 14:30 |
| mchiappero | or it's a bug in neutron.. ? | 14:30 |
| ltomasbo | what I hitted while working on the patches for the pool of ports | 14:30 |
| ltomasbo | is that if too many containers are created at once | 14:31 |
| mchiappero | (I hate neutron, this is a good chance to express it anyway :D) | 14:31 |
| ltomasbo | the neutron side gets kind of block | 14:31 |
| ltomasbo | I sometimes have to restart the openvswitch daemon | 14:31 |
| mchiappero | ok I'll add a sleep in the script | 14:31 |
| ltomasbo | otherwise the actions take forever | 14:31 |
| mchiappero | yeah I see :) | 14:31 |
| mchiappero | still stuck | 14:31 |
| mchiappero | thanks for the help man :) | 14:32 |
| ltomasbo | no problem | 14:32 |
| ltomasbo | if you find the problem is in kuryr, let me know! | 14:32 |
| mchiappero | but let me reiterate: Neutron sucks | 14:32 |
| mchiappero | :D | 14:32 |
| ltomasbo | to do my performance tests I restart both the ovs daemon and the neutron server and agent services | 14:32 |
| mchiappero | so far so good with VLANs other than this issue | 14:32 |
| ltomasbo | otherwise the results just get worse and worse over time | 14:33 |
| mchiappero | production quality!!! | 14:33 |
| ltomasbo | and this happens to me with both vlans and generic driver | 14:33 |
| yedongcan | apuimedo: ping | 14:33 |
| mchiappero | I'll keep it in mind, thanks | 14:33 |
| *** tianquan has quit IRC | 14:33 | |
| apuimedo | yedongcan: pong | 14:36 |
| yedongcan | apuimedo: not sure we can create only network with IPv6 data. | 14:36 |
| apuimedo | ltomasbo: I have a hunch that with dragonflow or ovn this blocking doesn't happen | 14:37 |
| yedongcan | apuimedo: docker network create --driver=kuryr --ipam-driver=kuryr --ipv6 --subnet fe80::/64 --gateway fe80::1 --ip-range fe80::/64 test_net2 | 14:37 |
| yedongcan | apuimedo: Is this a right format? | 14:37 |
| apuimedo | let me verify | 14:37 |
| ltomasbo | apuimedo, I would like to try it with OVN too. It is in my to do list... | 14:38 |
| apuimedo | it looks like it is | 14:39 |
| apuimedo | which error do you get yedongcan ? | 14:39 |
| yedongcan | apuimedo: In log info: Received JSON data {u'V6': False, u'Options': {}, u'Pool': u'', u'AddressSpace': u'no_address_space', u'SubPool': u''} for /IpamDriver.RequestPool ipam_request_pool | 14:41 |
| apuimedo | darmn | 14:42 |
| apuimedo | the bug is still there | 14:42 |
| apuimedo | yedongcan: which docker version? | 14:42 |
| yedongcan | Docker version 1.12.1, build 23cf638 | 14:42 |
| apuimedo | this was fixed for 1.13 | 14:43 |
| apuimedo | https://github.com/docker/docker/issues/28055 | 14:43 |
| yedongcan | ok, see it. | 14:44 |
| yedongcan | apuimedo: So how to deal with it? | 14:46 |
| * apuimedo looking | 14:46 | |
| apuimedo | yedongcan: which distro do you use? | 14:47 |
| yedongcan | ubuntu 14.04 | 14:48 |
| apuimedo | if you use the repos on get.docker.com, doesn't it have docker 1.13? | 14:49 |
| yedongcan | oops, let me check. maybe this is install manual. | 14:51 |
| yedongcan | I check it in my another env, docker version is 1.13. | 14:51 |
| apuimedo | yedongcan: so then I propose we add support for ipv6 only as well | 14:52 |
| apuimedo | and just put a reno release note saying that ipv6 only is for docker 1.13+ | 14:52 |
| *** mattmceuen has joined #openstack-kuryr | 14:53 | |
| yedongcan | apuimedo: yes, of course. I will check it and add that limit. | 14:53 |
| yedongcan | apuimedo: thanks. | 14:54 |
| apuimedo | thanks to you yedongcan! | 14:54 |
| apuimedo | great work with ipv6! | 14:54 |
| yedongcan | you're welcome, my pleasure. | 14:55 |
| apuimedo | :-) | 14:57 |
| mchiappero | ltomasbo: is there anything that can be done? | 15:02 |
| mchiappero | I'm actually unable to use thi stack anymore | 15:02 |
| mchiappero | *this | 15:02 |
| mchiappero | may I add a sleep somewhere? | 15:03 |
| ltomasbo | mchiappero, what I do is: | 15:03 |
| mchiappero | restarted everthing | 15:03 |
| ltomasbo | stop neutron server | 15:03 |
| mchiappero | still same crap | 15:03 |
| ltomasbo | restart openvswitch | 15:03 |
| ltomasbo | restart neutron server | 15:03 |
| ltomasbo | restrrat neutron-agent | 15:03 |
| ltomasbo | (in a devstack deployment) | 15:03 |
| mchiappero | did | 15:03 |
| mchiappero | I did | 15:04 |
| mchiappero | :( | 15:04 |
| ltomasbo | and then restart kuryr-controller (that one is not needed but I needed for the ports pool) | 15:04 |
| *** saneax is now known as saneax-_-|AFK | 15:04 | |
| ltomasbo | can you check if you have left-overs in the openvswitch with ovs-vsctl show? | 15:04 |
| mchiappero | do you mean dumping flows? | 15:05 |
| mchiappero | will the interface pooling solve the issue? | 15:09 |
| ltomasbo | not flows | 15:10 |
| ltomasbo | when I 'crash' the neutron server | 15:10 |
| ltomasbo | once I got a lot of ports left over at the ovs br-int | 15:10 |
| ltomasbo | and I had to cleaned up manually | 15:10 |
| *** hongbin has joined #openstack-kuryr | 15:12 | |
| *** tianquan has joined #openstack-kuryr | 15:12 | |
| mchiappero | ok | 15:12 |
| mchiappero | does anybody know whether there are threading libs in openstack? | 15:13 |
| mchiappero | or is using the plain python threading fine? | 15:13 |
| apuimedo | mchiappero: practically all of openstack uses green threads | 15:13 |
| ltomasbo | yep, green threads | 15:14 |
| apuimedo | eventlet | 15:14 |
| *** tianquan has quit IRC | 15:16 | |
| *** jchhatbar has joined #openstack-kuryr | 15:16 | |
| mchiappero | uhm, let me rephrase: s/threading/concurrency/ | 15:17 |
| mchiappero | I was referring to locks and co. | 15:17 |
| mchiappero | uhm I see semaphores | 15:17 |
| *** janki has quit IRC | 15:18 | |
| *** salv-orlando has joined #openstack-kuryr | 15:18 | |
| *** jchhatbar has quit IRC | 15:19 | |
| yedongcan | apuimedo: it seems 1.13 also meets the same problem. | 15:24 |
| *** salv-orlando has quit IRC | 15:24 | |
| mchiappero | OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') | 15:25 |
| mchiappero | restarting mysql will kill the stack right? | 15:26 |
| apuimedo | mchiappero: dunno | 15:27 |
| apuimedo | yedongcan: what does it send in the request? | 15:27 |
| yedongcan | apuimedo: docker network create --ipv6=true --driver=kuryr --ipam-driver=kuryr --subnet fe80::/64 --gateway fe80::1 --ip-range fe80::/64 test_net1 | 15:28 |
| yedongcan | DEBUG kuryr_libnetwork.controllers [-] Received JSON data {u'V6': False, u'Options': {}, u'Pool': u'', u'AddressSpace': u'no_address_space', u'SubPool': u''} for /IpamDriver.RequestPool ipam_request_pool | 15:28 |
| apuimedo | fucking hell | 15:29 |
| apuimedo | which package verison is tihs? | 15:29 |
| apuimedo | *this | 15:29 |
| yedongcan | docker version is 1.13.1 | 15:30 |
| mchiappero | ltomasbo: do you also delete the trunk bridge? | 15:30 |
| yedongcan | ubuntu 14.04 | 15:30 |
| ltomasbo | mchiappero, I did not have to do that | 15:31 |
| ltomasbo | but, perhaps it broke in a different way for you... | 15:31 |
| ltomasbo | still not working? | 15:31 |
| ltomasbo | perhaps you need to restart rabbitmq if the messages got stuck there | 15:32 |
| mchiappero | it works intermittently | 15:33 |
| mchiappero | I go through the restart cycle | 15:34 |
| mchiappero | and breaks soon after | 15:34 |
| mchiappero | I hate this thing | 15:34 |
| ltomasbo | :/ | 15:36 |
| mchiappero | ovs is clean | 15:41 |
| mchiappero | ok, I'll test macvlan for a while and see whether it works | 15:41 |
| *** garyloug_ has quit IRC | 15:42 | |
| *** garyloug_ has joined #openstack-kuryr | 15:44 | |
| *** mattmceuen has quit IRC | 15:47 | |
| ltomasbo | mchiappero, I'm getting the same crappy performance | 15:49 |
| ltomasbo | and I just realized that the agent log was eating 10 Gb and I'm running out of space | 15:50 |
| ltomasbo | maybe you can check if that is what is happening to you too | 15:50 |
| openstackgerrit | Dongcan Ye proposed openstack/kuryr-libnetwork master: Add support kuryr IPv6 subnet https://review.openstack.org/442294 | 15:54 |
| mchiappero | /opt/stack/logs in devstack, right? | 15:55 |
| mchiappero | uhm no, just 2.3 GB :D | 15:56 |
| mchiappero | 2017-03-08 15:55:43.393 29766 DEBUG kuryr_kubernetes.cni.api [-] CNI output: {'msg': 'timeout', 'cniVersion': '0.3.0', 'code': 200} _write_dict /opt/stack/kuryr-kubernetes/kuryr_kubernetes/cni/api.py:101 | 15:57 |
| mchiappero | same shit | 15:57 |
| mchiappero | well, it's a different one actually | 15:58 |
| *** dimak_ has quit IRC | 15:58 | |
| mchiappero | but kuryr is dead silent, I'm not sure where the problem is | 15:59 |
| apuimedo | yedongcan: I think we should get the upstream github issue reopened then | 15:59 |
| yedongcan | apuimedo: ok, I remember that you use fedora, does it works for you? | 16:02 |
| *** salv-orlando has joined #openstack-kuryr | 16:20 | |
| mchiappero | ltomasbo: by "crappy performance' you mean issue or throughput? | 16:22 |
| ltomasbo | not sure why, but what it takes ages to create a few ports | 16:23 |
| ltomasbo | and I see the neutron agent doing some agent rpc_lool iterations | 16:24 |
| ltomasbo | like 1000s of them | 16:24 |
| *** salv-orlando has quit IRC | 16:25 | |
| mchiappero | ltomasbo: I don't know, I only know that despite kuryr being a good and positive project we can't go to customers with these issues | 16:41 |
| mchiappero | BTW: does anybody know the cause of this: NetworkPlugin cni failed on the status hook for pod 'test-13-server-1' - Unexpected command output Device "eth0" does not exist | 16:44 |
| *** yedongcan has left #openstack-kuryr | 16:45 | |
| *** salv-orlando has joined #openstack-kuryr | 16:59 | |
| *** tianquan has joined #openstack-kuryr | 17:19 | |
| *** tianquan has quit IRC | 17:23 | |
| *** garyloug_ has quit IRC | 17:49 | |
| *** ltomasbo is now known as ltomasbo|away | 17:58 | |
| *** pcaruana has quit IRC | 19:13 | |
| *** salv-orl_ has joined #openstack-kuryr | 19:23 | |
| *** salv-orlando has quit IRC | 19:26 | |
| *** salv-orl_ has quit IRC | 19:37 | |
| *** tianquan has joined #openstack-kuryr | 20:22 | |
| *** tianquan has quit IRC | 20:26 | |
| *** salv-orlando has joined #openstack-kuryr | 20:38 | |
| *** salv-orlando has quit IRC | 20:42 | |
| *** yamamoto has joined #openstack-kuryr | 21:15 | |
| openstackgerrit | Marco Chiappero proposed openstack/kuryr-kubernetes master: [WIP] Add MACVLAN based interfaces for nested containers https://review.openstack.org/440669 | 21:16 |
| *** yamamoto has quit IRC | 21:18 | |
| *** yamamoto has joined #openstack-kuryr | 21:20 | |
| *** salv-orlando has joined #openstack-kuryr | 21:31 | |
| *** yamamoto has quit IRC | 21:47 | |
| *** tianquan has joined #openstack-kuryr | 22:24 | |
| *** tianquan has quit IRC | 22:28 | |
| *** yamamoto has joined #openstack-kuryr | 22:31 | |
| *** dougbtv has quit IRC | 22:37 | |
| *** tianquan has joined #openstack-kuryr | 22:45 | |
| *** tianquan has quit IRC | 23:16 | |
| *** saneax-_-|AFK is now known as saneax | 23:21 | |
| *** pmannidi has joined #openstack-kuryr | 23:27 | |
| *** david-lyle has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!