Monday, 2016-12-05

*** tonanhngo has joined #openstack-kuryr00:11
*** tonanhngo has quit IRC00:13
*** yuanying_ has joined #openstack-kuryr00:19
*** yuanying has quit IRC00:19
*** yuanying_ has quit IRC00:19
*** yuanying has joined #openstack-kuryr00:19
*** yuanying_ has joined #openstack-kuryr00:20
*** yuanying has quit IRC00:20
*** yuanying has joined #openstack-kuryr00:21
*** yuanying_ has quit IRC00:21
*** yuanying_ has joined #openstack-kuryr00:21
*** yuanying has quit IRC00:21
*** yuanying_ has quit IRC00:21
*** yuanying has joined #openstack-kuryr00:22
*** yuanying has quit IRC00:22
*** yuanying has joined #openstack-kuryr00:22
*** vikasc has quit IRC00:39
*** tonanhngo has joined #openstack-kuryr00:43
*** tonanhngo has quit IRC00:46
*** limao has joined #openstack-kuryr00:50
openstackgerritZhigang Li proposed openstack/kuryr-libnetwork: Replace six.iteritems() with .items()  https://review.openstack.org/40105601:10
*** tonanhngo has joined #openstack-kuryr01:11
*** tonanhngo has quit IRC01:13
*** shashank_hegde has quit IRC01:16
*** tonanhngo has joined #openstack-kuryr01:23
*** tonanhngo has quit IRC01:24
*** yedongcan has joined #openstack-kuryr01:29
*** shashank_hegde has joined #openstack-kuryr01:31
openstackgerritMerged openstack/fuxi: Retrieve volumes from cinder with matched metadata  https://review.openstack.org/39044801:40
*** tonanhngo has joined #openstack-kuryr02:02
*** tonanhngo has quit IRC02:04
*** hongbin has joined #openstack-kuryr02:10
*** tonanhngo has joined #openstack-kuryr02:21
*** tonanhngo has quit IRC02:23
*** tonanhngo has joined #openstack-kuryr02:35
*** tonanhngo has quit IRC02:37
openstackgerritdengshaolin proposed openstack/fuxi: Raise exception when find more than one matched Cinder volumes  https://review.openstack.org/38793702:44
*** vikasc has joined #openstack-kuryr02:47
openstackgerritdengshaolin proposed openstack/fuxi: Raise exception when create from existing volume which is being usd by another cluster  https://review.openstack.org/38792902:47
*** tonanhngo has joined #openstack-kuryr02:48
*** tonanhngo has quit IRC02:49
*** yuanying has quit IRC02:53
*** yuanying has joined #openstack-kuryr03:00
*** yamamoto_ has joined #openstack-kuryr03:02
*** shashank_hegde has quit IRC03:02
*** tonanhngo has joined #openstack-kuryr03:08
*** tonanhngo has quit IRC03:08
*** shashank_hegde has joined #openstack-kuryr03:09
*** limao has quit IRC03:16
*** limao has joined #openstack-kuryr03:17
*** tonanhngo has joined #openstack-kuryr03:21
*** limao has quit IRC03:21
*** tonanhngo has quit IRC03:22
*** dougbtv has quit IRC03:24
*** tonanhngo has joined #openstack-kuryr03:34
*** tonanhngo has quit IRC03:36
*** yuanying has quit IRC03:48
*** tonanhngo has joined #openstack-kuryr03:52
*** tonanhngo has quit IRC03:53
*** shashank_hegde has quit IRC03:53
*** tonanhngo has joined #openstack-kuryr04:05
*** tonanhngo has quit IRC04:07
*** yedongcan has quit IRC04:07
*** yedongcan has joined #openstack-kuryr04:08
*** tonanhngo has joined #openstack-kuryr04:18
*** tonanhngo has quit IRC04:20
*** tonanhngo has joined #openstack-kuryr04:31
*** tonanhngo has quit IRC04:33
*** tonanhngo has joined #openstack-kuryr04:47
*** limao has joined #openstack-kuryr04:48
*** tonanhngo has quit IRC04:48
*** yamamoto_ has quit IRC04:48
*** yamamoto_ has joined #openstack-kuryr04:50
*** limao_ has joined #openstack-kuryr04:51
*** limao has quit IRC04:52
*** yuanying has joined #openstack-kuryr04:54
*** tonanhngo has joined #openstack-kuryr05:00
*** tonanhngo has quit IRC05:01
*** yedongcan has quit IRC05:08
*** tonanhngo has joined #openstack-kuryr05:13
*** tonanhngo has quit IRC05:15
*** yedongcan has joined #openstack-kuryr05:17
*** saneax-_-|AFK is now known as saneax05:26
*** tonanhngo has joined #openstack-kuryr05:26
*** tonanhngo has quit IRC05:27
openstackgerritDongcan Ye proposed openstack/kuryr-libnetwork: Fix missing subnetpool_id in neutron subnet  https://review.openstack.org/38783505:33
*** janki has joined #openstack-kuryr05:34
*** tonanhngo has joined #openstack-kuryr05:39
*** tonanhngo has quit IRC05:40
openstackgerritDongcan Ye proposed openstack/kuryr-libnetwork: Fix missing subnetpool_id in neutron subnet  https://review.openstack.org/38783505:44
*** irenab has joined #openstack-kuryr05:50
*** tonanhngo has joined #openstack-kuryr05:52
*** tonanhngo has quit IRC05:53
*** tonanhngo has joined #openstack-kuryr06:06
*** tonanhngo has quit IRC06:07
apuimedohongbin: https://review.openstack.org/#/c/402244/406:08
apuimedoI posted a review after trying it out with my f2406:09
hongbinapuimedo: ack, let me check06:09
apuimedothanks06:09
* apuimedo going back to bed06:09
hongbinapuimedo: i tested it in ubuntu:14.0406:09
hongbinapuimedo: however, will test it in other distro as well06:10
hongbinapuimedo: thanks for the review06:10
*** tonanhngo has joined #openstack-kuryr06:19
*** tonanhngo has quit IRC06:20
*** hongbin has quit IRC06:21
*** tonanhngo has joined #openstack-kuryr06:32
*** yamamoto_ has quit IRC06:34
*** tonanhngo has quit IRC06:34
*** yamamoto_ has joined #openstack-kuryr06:36
*** tonanhngo has joined #openstack-kuryr06:45
*** tonanhngo has quit IRC06:46
*** shashank_hegde has joined #openstack-kuryr06:47
*** yuanying_ has joined #openstack-kuryr06:52
*** yuanying has quit IRC06:55
*** tonanhngo has joined #openstack-kuryr06:58
*** tonanhngo has quit IRC06:59
*** limao_ has quit IRC07:08
*** yedongcan has quit IRC07:08
*** tonanhngo has joined #openstack-kuryr07:11
*** tonanhngo has quit IRC07:13
*** oanson has joined #openstack-kuryr07:16
openstackgerritMerged openstack/kuryr: Delete deprecated Hacking in tox.ini  https://review.openstack.org/40654507:16
irenabapuimedo: ivc_ ping07:18
*** yedongcan has joined #openstack-kuryr07:19
*** dimak has joined #openstack-kuryr07:44
*** limao has joined #openstack-kuryr07:50
*** limao has quit IRC08:01
*** openstackgerrit has quit IRC08:03
*** yuanying_ has quit IRC08:19
*** dimak has quit IRC08:26
apuimedoirenab: pong08:28
irenabapuimedo: good morning08:29
apuimedoirenab: boker tov08:29
irenabapuimedo: :-)08:29
irenabapuimedo: shortquestion on devstack with k8s08:30
apuimedogo ahead08:30
irenabI deployed the one with ivc patch (quite sure it isnot related), and stack.sh clains to fail since cannot start docker service. It is already running. In general everything seems to be working, so not sure if its a bug. did you see the same?08:31
apuimedoirenab: that's an unstack bug IIRC08:31
apuimedosometimes it happens, and the next stacking goes to hell08:32
apuimedoirenab: mind filing a bug. I'll improve it08:32
apuimedoas a workaround. Just unstack and manually uninstall docker08:32
irenabapuimedo:  actually it seems that all other services are fine, thats why I am not sure how to describe the bug08:32
apuimedoirenab: ok. I'll investigate a bit more, usually I just work around it :P08:34
irenabapuimedo: just not sure ifsuch setup is expected to work. so to workaround you justkull running docker and run the one from devstack screen?08:36
*** dimak has joined #openstack-kuryr08:37
*** shashank_hegde has quit IRC08:38
apuimedoirenab: I usually stack again08:38
apuimedobecause it will need to launch the other containers08:38
apuimedoetcd, k8s, etc08:38
irenabanother issue I saw is that kubelet is not started since it requires nsenter08:39
irenabhad to add it manually08:39
apuimedoirenab: that's very odd. It started for me08:46
apuimedoI saw it in kubectl get nodes08:46
apuimedowhich distro do you use for devstack?08:46
irenabubuntu14.0408:47
irenabactually I too the vagrant from the libnetwork and just reduced a bit from it08:47
*** palexster has quit IRC08:50
*** lmdaly has joined #openstack-kuryr08:51
*** palexster has joined #openstack-kuryr08:55
*** limao has joined #openstack-kuryr08:57
apuimedoaha08:59
apuimedoI used a new f2408:59
irenabapuimedo: so on f24 nsenter was already installed?09:00
irenabI will fill a bug for now09:01
*** irenab_ has joined #openstack-kuryr09:05
*** ram_____ has joined #openstack-kuryr09:05
apuimedoI suppose09:05
*** irenab has quit IRC09:06
*** irenab_ is now known as irenab09:06
apuimedoirenab: yes. It seems so. It was provided by util-linux-2.28.2-1.fc24.x86_6409:06
*** yuanying has joined #openstack-kuryr09:08
ram_____apuimedo: Hi. I am trying to deploy basic devstack- kuryr. But I was facing issue. Issue info: http://paste.openstack.org/show/591390/. Can you please help me in this.09:09
*** openstackgerrit has joined #openstack-kuryr09:11
openstackgerritOpenStack Proposal Bot proposed openstack/fuxi: Updated from global requirements  https://review.openstack.org/37374509:11
* apuimedo checking, ram_____09:11
apuimedoram_____: I'll make a new deployment to verify09:12
yedongcanlimao: ping09:13
limaoyedongcan: hi09:14
ram_____apuimedo : Thanks. I have taken Ubuntu 16.0409:14
yedongcanlimao: I had updated the patch today, you can review it if you have free time.09:15
limaoyedongcan: sure, thanks09:15
yedongcanlimao: thanks.09:15
yedongcanapuimedo, irenab: Hi , what's your opinions of this patch: https://review.openstack.org/#/c/404591/09:18
irenabyedongcan: will check in 40 mins, currently at the meeting09:18
yedongcanirenab: thanks09:19
*** reedip has quit IRC09:22
*** garyloug has joined #openstack-kuryr09:24
apuimedoyedongcan: does it happen on 16.04 or only on 14.04?09:24
yedongcanapuimedo: the gate is running on 14.0409:28
yedongcanapuimedo: http://logs.openstack.org/35/387835/4/check/gate-kuryr-libnetwork-dsvm-rally-nv/0576c1f/09:29
apuimedoyedongcan: maybe we could try if it happens on 16.04 and move it to that09:30
yedongcanapuimedo: sure, we can try it in local, but we can't update gate machine.09:32
apuimedoyedongcan: but we can change which node is used to one with 16.06 IIRC09:33
irenabapuimedo: is there any ‘official’  distro versions that kuryr is supposed to work on? It should be the same as openstack, right?09:36
yedongcanapuimedo: 14.04 sometimes can run successful, http://logs.openstack.org/65/400365/7/check/gate-kuryr-libnetwork-dsvm-rally-nv/06789a7/console.html09:38
apuimedo16.04 and f24 are the ones I care the most09:38
apuimedopre systemd... not so much09:39
apuimedoirenab: ^^09:39
irenabapuimedo: got it, thanks. Probably we should bump the vagrant versions09:40
apuimedowe should09:40
* apuimedo doesn't use vagrant09:40
apuimedo /q vikasc09:43
apuimedovikasc: ping09:43
vikascapuimedo, pong09:43
vikascapuimedo, u have a very long life09:43
apuimedo?09:43
vikascapuimedo, just now was thinking to ping u09:43
apuimedoI replied to one of the comments you put on ivc_'s cni patch09:44
vikascapuimedo, just saw09:44
apuimedo;-)09:44
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-libnetwork: Nested-Containers: trunk subports management  https://review.openstack.org/40246209:56
*** pmannidi has quit IRC09:59
*** reedip has joined #openstack-kuryr10:00
*** tonanhngo has joined #openstack-kuryr10:01
*** tonanhngo has quit IRC10:03
apuimedoram_____: I just did a redeploy10:09
apuimedoand it ended successfully10:09
apuimedoram_____: you may have been unlucky when pulling neutron-lib and neutron10:10
apuimedofor reference, my local.conf is10:10
apuimedoram_____: https://paste.fedoraproject.org/499599/14809327/10:12
*** limao has quit IRC10:14
ram_____apuimedo: Oh. Thank you very much. I got the point. I used neutron plugin extra. I will try with your conf.10:16
apuimedothanks10:16
apuimedoirenab: ivc_: The more I think about it, the more I want to use k8s' third party resources for making neutron notifications end up in k8s and just have normal k8s watchers instead of polling neutron10:18
*** yedongcan has left #openstack-kuryr10:24
*** irenab has quit IRC10:25
*** yamamoto_ has quit IRC10:27
*** gsagie has joined #openstack-kuryr10:27
*** yamamoto has joined #openstack-kuryr10:43
*** irenab has joined #openstack-kuryr10:50
irenabapuimedo: do you want neutron notifier to post on k8s-api?10:50
apuimedoirenab: That's what I'm considering10:51
ivc_apuimedo do you mean that we would replace 'annotations' with ThirdPartyResource with custom fields? or keep 'annotations' but move them to ThirdPartyResource?10:51
apuimedoivc_: I'm still in that10:51
apuimedodeciding if we do a mixed approach10:51
apuimedos/if we do/if I propose/10:52
apuimedoreplacing completely would be problematic10:52
irenabapuimedo: for mixed approach, you mean daemon +   slim CNI?10:52
ivc_apuimedo, irenab, i'd prefer if instead of using neutron notifier we make another daemon that will use neutron's MQ and translate events into ThirdPartyResource10:52
apuimedoirenab: ivc_: I mean mixed as in, some things go in normal annotations, some others, like lbaasv2 listeners and pools go into third party resources10:53
irenabivc_: apuimedo : need to think a bit on this. I am still recovering from illness, so thinking may take a while:-)10:53
apuimedoivc_: I don't really care all that much, what is the source that ends up putting stuff on third party resources10:53
ivc_i mean in neutron we have 2 facilities - notifier (same as nova; could be problematic to implement as it affects neutron codebase) and notifications (used by ceilometer and imo designed for our case)10:53
apuimedoirenab: the trick is not recovering10:53
* apuimedo still with a big cold10:53
apuimedoxD10:53
apuimedoivc_: I know10:53
apuimedoI'm now checking what's on octavia's, now that the neutron lbaas agent moved there10:54
apuimedodepending on what's there, it may not even be necessary to take a decision10:54
apuimedobecause maybe it was taken for us already10:55
ivc_apuimedo what 'decision' are you talking about here? xD10:55
apuimedoivc_: notifier vs rpc10:56
apuimedoit's likely that octavia only has the rpc10:56
ivc_notifier is for nova and port activity only10:56
apuimedoso nova notifier most likely just goes out of the window10:56
apuimedoexactly10:56
irenabapuimedo: if lbas is separate project, it should not have RPC channel with neutron, only use its ‘official’ apis10:56
apuimedoirenab: IIUC both neutron and lbaas publish to rpc10:56
irenabREAST + notifications10:57
apuimedowe'll have to subscribe to both10:57
apuimedoirenab: reast?10:57
irenabREST10:57
ivc_we could have rpc (tho i don't think rpc is the right term here, its just MQ notification) as primary choice and polling as fall-back10:57
apuimedoah10:57
irenabmy understanding is that RPC is internal channel between neutron servers and neutron agents10:58
ivc_irenab, thats why i say RPC is not the right term here :)10:58
apuimedoivc_: short term makes sense. Long term I really prefer that the watcher only needs to check resources and third party resources10:58
apuimedofor events10:58
irenabivc_: by MQ you amly that neutron will have to implement something?10:58
ivc_we are talking about http://docs.openstack.org/admin-guide/networking-adv-operational-features.html10:58
apuimedoirenab: ivc_: sorry. i meant MQ notifications10:58
irenabthe ones that ceilometer use?10:59
ivc_apuimedo, irenab, the link above is about MQ notifications (used by ceilometer)10:59
ivc_note that it is not enabled by default and it would be nice to support polling as fallback imo11:00
apuimedoif it's not default, we may very well have to have a fallback11:01
irenabapuimedo: ivc_ :so to catch up on your proposal, you suggest to add 3rd party resource that will be created/updated onceproper notification is received?11:02
irenabor polling done11:02
apuimedoirenab: that's what I'm thinking about, yes11:02
apuimedothat listener and pool notifications will end up in third party resources11:02
irenabapuimedo: what is the advantage versus annotation approach?11:03
irenabapuimedo: is it only for service support oralso for pod?11:03
apuimedoirenab: we could just add more data into the service annotation, of course, I just think that it is probably not relevant11:03
apuimedobut!11:04
apuimedoI may be wrong11:04
apuimedoand we may end up doing it in annotations11:04
irenabapuimedo: ivc_ I probably missed when you started service discussion, feeling a bit in the middle11:04
apuimedoand putting listeners on the services annotation and pool in the endpoints annotation11:04
apuimedoirenab: heh... We didn't start... That was me just having to think instead of type due to holding the baby a while this morning11:05
irenabapuimedo: :-)11:05
apuimedoin any case, the important part here is. I'd really prefer if we can have another component to post/update K8s resources (or thirdparty resources) and the watcher process only needs to 'watch'11:06
irenabapuimedo: so you want the processwide split, right? since from the SW perspective, watcher invokes handlers,so there is sort of separation11:07
irenabivc_ has incredibly generic model there11:07
apuimedoirenab: well, it's not really splitting the process in that axis11:08
apuimedoit's adding another daemon that receives notifications11:08
apuimedoand patches resources11:08
apuimedocompletely orthogonal and independent to the watches/handlers11:08
irenabapuimedo: so daemin for read and daemon for write?11:09
irenabadditional daemon only for notifications or polling too?11:09
apuimedoirenab: I would make the same daemon for both approaches11:10
apuimedothink of it as a MQ driver and a polling driver11:10
apuimedoand that in the end it always goes to K8s to update some resource11:10
irenabI am not experienced with Octavia, but probably pool lifecycle has states, so you need to wait till it becomes Active, right?11:10
*** reedip has quit IRC11:10
apuimedoirenab: I'm trying to find out if they send MQ notifications11:10
apuimedofor their resources11:10
irenaband you prefer to keep this async?11:11
apuimedoif they don't, we have to fallback to polling11:11
apuimedowith disappointment11:11
irenabapuimedo: what is the main drive for separate daemon?11:12
irenabkeep it simple?11:12
apuimedoirenab: deployment11:12
apuimedoand simplicity11:12
apuimedoI don't want temptations to couple things afterwards11:12
irenabjust more to take care for HA11:13
irenabbut may be reasonable approach, expecially with all the micro services trend11:14
apuimedothat's ture11:14
apuimedo*true11:14
apuimedoit is a tradeoff indeed11:14
*** reedip has joined #openstack-kuryr11:24
apuimedoirenab: ivc_: Octavia has an Event Streamer that is stevedore based11:24
irenabapuimedo: nice11:24
apuimedothis means that we could probably have our own driver for it11:24
apuimedothat goes straight to k8s11:24
irenabso either need to know what object to patch or create new one11:25
apuimedoirenab: https://github.com/openstack/octavia/blob/master/octavia/controller/queue/endpoint.py11:25
apuimedoirenab: well, my idea of the thirdparty resources was the following11:26
apuimedofrom the services handler, when you create, let's say a listener11:26
apuimedoyou know its UUID11:26
apuimedothen you can watch for a thirdparty resource with that UUID as name11:27
apuimedofrom the stevedore driver11:27
apuimedo(this would not be necessary if octavia objects had tags, obviously11:27
*** reedip has quit IRC11:28
apuimedoit seems it may only be for the health manager11:33
apuimedoI don't know if it always run11:33
openstackgerritvikas choudhary proposed openstack/kuryr-libnetwork: Delay neutron extension checks to first neutron interaction  https://review.openstack.org/40332511:36
apuimedodevstack seems to run it11:36
irenabapuimedo: wecan do the same as with networkbefore tags, use its name11:39
apuimedoirenab: maybe11:39
*** reedip has joined #openstack-kuryr11:41
irenabneed to check the model, hope listener has name attribute11:41
apuimedoirenab: it does11:42
irenabapuimedo: anyway, its just implementation details. with octavia notifier it is possible to make clean separation11:43
apuimedoirenab: it even has description :-)11:44
apuimedo(and l7 policies that may be usable in the future for ingress controller)11:45
irenabapuimedo: was also thinking about it :-)11:46
apuimedo•_•)11:47
apuimedo( •_•)>⌐■-■,11:47
apuimedo(⌐■_■)11:47
gsagiesnow man?11:48
irenabwith a gun?11:48
*** gsagie has quit IRC11:49
apuimedono!11:50
apuimedoirenab: https://33.media.tumblr.com/8d4adac905b4c9155ba64a4a2b8ef6ca/tumblr_nbam09PAtE1spxz7uo1_250.gif11:51
irenab:-)11:51
apuimedomore inspired on https://media.giphy.com/media/v9rfTQBNqdsSA/giphy.gif11:51
*** limao has joined #openstack-kuryr11:58
*** limao has quit IRC12:03
ivc_apuimedo, irenab, sry was on a meeting for the past hour, so wasn't able to respond. i think using k8s 3rd party resources makes a lot of sense for both lbaas and ports12:03
irenabivc_: trying out your latest patch12:04
ivc_that allows us to have full control over them. might also solve the current 'leftover hybrid bridge' problem we have now due to cni being called after pod is deleted from k8s db12:05
irenabivc_: if I want more detals on k8s service log, what flag do I add?12:05
ivc_irenab, uhm, what? :)12:05
ivc_you mean from cni?12:06
irenabnow, the controller12:06
irenabI see port being created and active, but kubelet fails12:07
irenabdocker_manager.go:357] NetworkPlugin cni failed on the status hook for pod 'my-nginx-379829228-z1at9' - Unexpected command output Device "eth0" does not exist.12:07
ivc_for the controller just add '-d' after 'run_server.py'12:08
ivc_for the cni, add "'debug': true" to 10-kuryr.conf12:08
ivc_you probably want the 'cni' debug12:09
irenabivc_: probably both. thanks12:10
*** limao has joined #openstack-kuryr12:11
*** limao has quit IRC12:21
*** limao has joined #openstack-kuryr12:21
ivc_irenab, are you stacking on a clean env? also do you 'kubectl run' with a single replica?12:22
irenabivc_: now run again stack.sh, previously single pod wasnot running, but saw neutron port created for it with Active status12:23
irenabivc_:  something strange. I just had new stack completed, but see kubelet comlain on not being able to plug pod12:25
ivc_irenab, kk. thats weird actually, by the time it becomes 'active' veth should already be created. if it happens again, pls try 'kubectl exec ... ip a' and check if 'eth0' is there12:26
ivc_irenab, apuimedo, vikasc, i can confirm that https://review.openstack.org/#/c/404038/6/devstack/plugin.sh@79 is indeed broken12:27
apuimedoivc_: so that means that it should the link should be done on postconfig12:30
ivc_apuimedo yup, or we can hardcode '/usr/local/bin/kuryr-cni' instead of 'which'12:31
apuimedoivc_: nope. It's distro dependant12:31
ivc_true12:31
apuimedoand the path you put is a result of debian's/Ubuntu's screwed patching of distutils12:31
apuimedoit always bothered me that they change distutils to point to local12:32
ivc_:)12:32
apuimedoI have some python code to detect it, if you want12:32
apuimedoso instead of which, this oneliner prints the path that it will have12:32
ivc_i actually like it that they use /usr/local12:32
apuimedowhen I'm root and I install with pip it pisses me off12:33
ivc_well, /usr/bin is for .deb and /usr/local for custom stuff. makes sense to me :)12:33
apuimedoI'm not saying it doesn't make sense, I say I don't like it12:34
apuimedomaybe if pip supported parameters like --prefix12:34
apuimedo:P12:34
*** limao has quit IRC12:36
irenabivc_: http://paste.openstack.org/show/591412/12:38
irenabivc_: log message on kubelet is terrible, very difficult to find the issue with tons of content12:40
irenabivc_: intersting that the same logging blockis on pod describe12:45
*** limao has joined #openstack-kuryr12:51
ivc_irenab, i think i've seen those '\n' as real linebreaks in my logs. i'm restacking now12:51
ivc_irenab, do you use ubuntu or centos for devstack?12:52
irenabubuntu 14.0412:52
irenabshall I move to 16.04?12:52
ivc_i'm using 16.04; it should not matter tbh12:53
*** garyloug has quit IRC12:54
ivc_i'm a bit worried about 'describe' those log messages should go to stderr and for 'describe' i would expect the CNI-formatted stdout errors12:54
*** dougbtv has joined #openstack-kuryr12:54
irenabas for the kubelet failure, it seems to be at https://review.openstack.org/#/c/404038/6/kuryr_kubernetes/cni/binding/bridge.py line 36-3712:55
irenabivc_: yes, in ‘pod describe’  it looks very unexpeted , and appears few times12:56
irenabivc_: looks like this  http://paste.openstack.org/show/591414/12:58
ivc_irenab, yeah. it looks like we got oslo_log in stdout instead of stderr12:59
ivc_irenab, now i wonder how did it work for me http://docs.openstack.org/developer/oslo.log/opts.html#DEFAULT.use_stderr13:01
ivc_as it is stdout by default >.<13:02
ivc_apuimedo btw, symlink creation is done in 'post-config', does that mean that 'post-config' is called before 'install'?13:03
ivc_apuimedo i'll just move symlink to 'install' anyway13:04
*** reedip has quit IRC13:11
ivc_apuimedo in fact, 'ln -s' fails not due to 'which', but due to the lack of 'CNI_BIN_DIR'13:11
*** lmdaly has quit IRC13:12
apuimedoivc_: shouldn't13:12
*** limao has quit IRC13:12
apuimedoheh13:12
ivc_CNI_BIN_DIR is created in your 'extract_hyperkube'13:12
apuimedoivc_: Maybe we should use install instead of symlink13:12
*** limao has joined #openstack-kuryr13:12
*** irenab has quit IRC13:13
ivc_apuimedo but use 'which'?13:13
apuimedoI don't mind ending with two copies13:13
apuimedoin the end they will only call the library in /opt/stack/kuryr-kubernetes13:13
ivc_apuimedo, irenab, we need fullstack gates ...13:23
*** reedip has joined #openstack-kuryr13:24
*** yamamoto has quit IRC13:26
*** janki has quit IRC13:31
*** limao_ has joined #openstack-kuryr13:34
*** irenab has joined #openstack-kuryr13:35
*** limao has quit IRC13:36
apuimedoivc_: I have to fix my patch :/13:37
ivc_apuimedo what?13:37
ivc_apuimedo oh, the one for gate you mean?13:38
apuimedoivc_: yes. The one that runs the installation13:39
apuimedoand then, after your patch13:39
apuimedoI'll make one that tries starting to containers13:39
ivc_apuimedo, irenab, i've fixed issues with logs and ln. gonna push for review now. just need to verify once more13:40
*** yamamoto has joined #openstack-kuryr13:44
apuimedoivc_: I'll check it too when you push13:45
apuimedoivc_: irenab: I'd like to merge it today13:45
*** gsagie has joined #openstack-kuryr13:46
*** lmdaly has joined #openstack-kuryr13:49
apuimedoltomasbo: do the vlan devices remain still or are they properly removed?13:51
apuimedowith https://review.openstack.org/#/c/402462/13:51
ltomasbohi apuimedo13:52
ltomasboin meeting right now, so I may answer with delay13:52
ltomasbobut yes, they work fine right now13:52
ltomasboI included the driver detach in the patch as irenab suggested13:53
ltomasbobut now it seems it is colliding with the other patch which also splits the port binding drivers out13:53
ivc_irenab, apuimedo: here's what i get in kubelet.log: http://paste.openstack.org/show/591429/13:53
ltomasbonot sure about the proper order of both patches13:53
*** limao_ has quit IRC13:54
ivc_irenab, apuimedo seems to work, but not sure how to interpret: E1205 05:49:27.553765  125394 docker_manager.go:357] NetworkPlugin cni failed on the status hook for pod 'nginx-701339712-aj9pg' - Unexpected command output Device "eth0" does not exist.13:54
ivc_ with error: exit status 113:54
*** limao has joined #openstack-kuryr13:55
apuimedoivc_: and the device is there?13:56
ivc_yup13:56
apuimedoivc_: you got nsenter?13:56
ivc_yes13:57
ivc_apuimedo, irenab check L111-112 in http://paste.openstack.org/show/591429/13:57
apuimedoaha!13:58
apuimedoeven worse13:58
apuimedoI wonder WTH they use ethtool13:58
ivc_what bothers me is that 'cni error' is BEFORE our cni is called13:58
apuimedoyou mean line 83?13:59
ivc_yes13:59
ivc_check timestamps in L83 and L8613:59
* apuimedo checking13:59
ivc_L86 is stderr of our cni13:59
ivc_so i wonder WTF that L83 is14:00
* apuimedo checking k8s code14:00
openstackgerritIlya Chukhnakov proposed openstack/kuryr-kubernetes: Experimental CNI & VIFBridge binding  https://review.openstack.org/40403814:01
ivc_apuimedo, irenab, done: https://review.openstack.org/#/c/404038/6..714:02
irenabivc_: checking14:03
ivc_also updated 10-kuryr.conf to include 'debug': true by default14:03
*** yedongcan has joined #openstack-kuryr14:03
mchiapperomeeting time?14:03
ivc_imo makes sense while we are in 'development' phase14:03
mchiappero:P14:03
irenabivc_: need to restack or restart k8s-kuryr and kubelet is enough?14:03
ivc_yup14:03
*** tonanhngo has joined #openstack-kuryr14:03
ivc_irenab, restart should do it14:03
*** tonanhngo has quit IRC14:04
ivc_but make sure you got /opt/stack/cni/bin/kuryr-cni14:04
*** garyloug has joined #openstack-kuryr14:04
*** oanson has quit IRC14:04
ivc_and run 'ln -s $(which kuryr-cni) /opt/stack/cni/bin/kuryr-cni' if you dont14:04
irenabI have14:05
*** hongbin has joined #openstack-kuryr14:09
irenabivc_: still get the trace in the ‘describe pod’14:12
ivc_irenab, oh you also need '[DEFAULT]use_stderr=true' in /etc/kuryr/kuryr.conf14:13
irenabivc_: got it,  thanks14:13
ivc_plugin has it so restacking would have updated it14:13
irenabjust restarted the services14:13
ivc_that's not for controller (tho it affects it) but just for CNI14:14
*** limao has quit IRC14:16
*** limao has joined #openstack-kuryr14:16
ivc_lol i've run '--replicas=40' and it killed OVS :) another reason to have daemon to limit the DoP :)14:18
ivc_(thats unrelated to kuryr-k8s)14:19
apuimedoivc_: what do you mean 'it killed ovs' ?14:19
apuimedowhich part went to hell?14:19
apuimedothe daemon?14:19
apuimedo!14:19
ivc_my vm is 99% in systime :)14:19
apuimedo:O14:19
apuimedothat doesn't sound good14:19
ivc_ovs-vsctl show, kubectl get pods, neutron port-list14:19
ivc_neither of them work :)14:19
ivc_i'm pretty sure i could do the same with nova14:19
janonymous:D sounds interesting :P14:20
ivc_well maybe i cant do it with nova as nova is slow :)14:20
apuimedoxD14:21
apuimedobut come on, ovs must have tests that storm it much more14:22
ivc_i love finding weak points and hammering them xD14:22
apuimedosomething must have happened14:22
ivc_apuimedo, neutron-agent has quite a bit of ERRORs too :)14:24
apuimedothat will be relevant to ajo's interests :-)14:25
apuimedoivc_: some example?14:25
ivc_2016-12-05 06:24:33.919 119247 ERROR oslo.messaging._drivers.impl_rabbit [-] [31ad0a5f-d6b3-4aeb-9e0b-698a1f3f8f7b] AMQP server 172.16.240.46:5672 closed the connection. Check login credentials: Socket closed14:25
ivc_2016-12-05 06:24:42.324 119247 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [req-86cfb37b-5bb6-4714-a7a1-63c5afe1de11 - -] Bridge br-int changed its datapath-ID from e668938d8946 to 0000e668938d894614:25
ivc_2016-12-05 06:23:42.074 119247 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-86cfb37b-5bb6-4714-a7a1-63c5afe1de11 - -] Switch connection timeout14:26
ivc_first q-agt error:14:28
ivc_2016-12-05 06:09:23.420 119247 ERROR neutron.common.rpc [req-86cfb37b-5bb6-4714-a7a1-63c5afe1de11 - -] Ti14:28
ivc_meout in RPC method update_device_list. Waiting for 1 seconds before next attempt. If the server is not d14:28
ivc_own, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unabl14:28
ivc_e to respond quickly enough.14:28
ivc_2016-12-05 06:09:24.059 119247 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req14:28
ivc_-86cfb37b-5bb6-4714-a7a1-63c5afe1de11 - -] Error while processing VIF ports14:28
apuimedoivc_: must be a timeout trying to get things from ovs14:28
ivc_apuimedo could be14:28
ivc_apuimedo but hey we only created 40 ports, right? :)14:29
apuimedoright14:29
apuimedowhat I mean is, I think the ovs agent errors are a symptom14:29
apuimedonot the root cause14:29
ivc_i think what causing this is the the storm on NETLINK from 40 CNI's IPDBs14:30
apuimedo40 doesn't sound like much14:31
apuimedofor netlink14:31
ivc_apuimedo 2016-12-05 06:08:23.340 128463 DEBUG pyroute2.netlink.nlsocket [-] Packet burst: delta=13 qsize=13 delay=0.01 get /usr/local/lib/python2.7/dist-packages/pyroute2/netlink/nlsocket.py:71614:33
ivc_:)14:33
ivc_got a bunch of those14:34
*** limao has quit IRC14:37
*** limao has joined #openstack-kuryr14:37
apuimedoivc_: maybe svinota can give us some feedback on that14:38
irenabivc_: still having issues to run single pod. did you try on ubuntu 14.04?14:38
apuimedoirenab: you are doing archeology with that distro :P14:39
ivc_irenab, nope, only on 16.04. ok i'll give it a try on a fresh 14.0414:39
apuimedoxD14:39
apuimedoirenab: which errors do you get?14:39
apuimedoI'll try reproducing later14:39
irenabapuimedo: just wanted to run devstack and try the patch14:39
apuimedoirenab: I know14:40
apuimedoI was just making fun of trusty14:40
apuimedoirenab: did you post some pastebin above?14:40
irenabyes14:40
irenabivc_: will try to run 16.04 later today/tomorrow morning and post +2 onceworks14:44
ivc_kk. i'll see if it works on clean 14.04; maybe you got some issues with some leftovers from initial stack/unstack14:45
apuimedoivc_: could be. I need to improve devstack to be a bit more corrosive14:47
ivc_irenab, apuimedo, those UML, BPMN, MDA, SysML in https://www.modelio.org feel so enterprise-y14:47
apuimedoivc_: Enterprise ready™14:47
apuimedoxD14:47
* ivc_ looks for apuimedo 'suicide' smiley14:51
* ivc_ y=ー( ゚д゚)・∵.14:53
apuimedo:D14:54
apuimedoyou found it!14:54
ivc_:P14:54
apuimedo(屮゚Д゚)屮14:54
apuimedoI have a few aliases in weechat :D14:54
*** limao has quit IRC14:58
*** yedongcan has left #openstack-kuryr14:58
*** limao has joined #openstack-kuryr14:59
ivc_apuimedo, irenab btw, note L86-L110 in http://paste.openstack.org/show/591429/ - its just 4 seconde!!!15:01
*** hongbin has quit IRC15:02
apuimedowth15:02
apuimedothat's terrible15:02
apuimedoxD15:02
ivc_apuimedo, yeah, last week it was taking anything from 25 to 60 seconds15:12
ivc_i wonder if i pulled some q-agt patch :)15:12
ivc_or maybe i just got lucky with some some well aligned timings15:13
apuimedoivc_: did we cause any trouble on k8s with the 40 watch cni processes?15:14
ivc_apuimedo the whole vm went nuts, so yes :)15:15
apuimedo:P15:16
apuimedohow big was it?15:16
ivc_8g/2cpu15:16
ivc_only 3g used15:17
*** saneax is now known as saneax-_-|AFK15:17
ivc_but the fun thing is that its working :) it was 1/40 at start, now 1 hour later its 4/40 :)15:18
*** limao has quit IRC15:19
*** reedip has quit IRC15:20
*** limao has joined #openstack-kuryr15:20
*** gsagie has quit IRC15:21
apuimedo:O15:22
apuimedothat's ludicrous speed15:23
ivc_its ok :) you can hardly expect anything when OVS is not responding well15:24
ivc_but i'm puzzled as last week it was slow, but no issues with OVS, but this week its fast but has issues with OVS. gonna do another clean restack on both 14.04 and 16.04 to see if it is consistent15:25
apuimedohttps://www.youtube.com/watch?v=ygE01sOhzz015:25
ivc_xD15:26
*** hongbin has joined #openstack-kuryr15:31
*** reedip has joined #openstack-kuryr15:32
*** oanson has joined #openstack-kuryr15:35
*** limao has quit IRC15:41
*** limao has joined #openstack-kuryr15:41
*** dougbtv_ has joined #openstack-kuryr15:59
*** dougbtv has quit IRC15:59
*** dougbtv_ is now known as dougbtv15:59
*** dougbtv_ has joined #openstack-kuryr16:00
*** limao has quit IRC16:02
*** limao has joined #openstack-kuryr16:02
*** yamamoto has quit IRC16:19
*** limao has quit IRC16:23
*** limao has joined #openstack-kuryr16:24
mchiapperoI just had a very quick look at the vlan driver, I would say there is a lot in common, I'll have another more in depth look later but I think we need to combine those changes somehow16:25
mchiapperoI would use classes as you ltomasbo did, but I would primarily use our interface and see how to extend that so that can fit your driver too16:26
mchiapperoBut I guess we would need to either combine them and abandon one, or simply start with one and drop some pieces from the other16:27
*** yuanying has quit IRC16:29
*** dimak has quit IRC16:33
*** janonymous has quit IRC16:34
ltomasbomchiappero, I agree they have a lot in common16:39
ltomasbothe main different I see is that yours in just focused in the port binding, while mine has also to do with some extra actions that needs to be done depending on the driver used16:40
ltomasbobut I agree they can be merge to try to cover both in just one16:41
ltomasbomchiappero, let me know when you take a deeper look if you think it is easy to merge both driver models (perhaps inside my patch as it also cover the vlan kuryr-libnetwork functionality)16:42
*** reedip has quit IRC16:43
mchiapperoone approach, that is disruptive for both is to have the base system first and then the drivers added16:45
mchiapperolater16:45
mchiapperoor some drivers added later16:45
mchiapperobut we need the interface we have (I agree with moving to classes)16:45
mchiapperoI guess you have less requirements there, I'll have a look later16:45
ltomasbook great16:46
*** limao has quit IRC16:47
*** irenab has quit IRC16:50
*** irenab has joined #openstack-kuryr16:54
openstackgerritDarla Ahlert proposed openstack/fuxi: Move from "tenant" to "project"  https://review.openstack.org/40618916:55
*** reedip has joined #openstack-kuryr16:56
*** irenab_ has joined #openstack-kuryr17:00
*** irenab has quit IRC17:01
*** irenab_ is now known as irenab17:01
*** reedip has quit IRC17:02
openstackgerritIlya Chukhnakov proposed openstack/kuryr-kubernetes: Experimental CNI & VIFBridge binding  https://review.openstack.org/40403817:02
*** dougbtv_ has quit IRC17:07
*** reedip has joined #openstack-kuryr17:15
*** yamamoto has joined #openstack-kuryr17:20
*** yamamoto has quit IRC17:26
*** lmdaly_ has joined #openstack-kuryr17:31
*** lmdaly has quit IRC17:31
*** yuanying has joined #openstack-kuryr17:38
*** shashank_hegde has joined #openstack-kuryr18:02
openstackgerritIlya Chukhnakov proposed openstack/kuryr-kubernetes: devstack: add nsenter and fix setup-files race  https://review.openstack.org/40711918:03
openstackgerritIlya Chukhnakov proposed openstack/kuryr-kubernetes: Experimental CNI & VIFBridge binding  https://review.openstack.org/40403818:05
*** reedip has quit IRC18:08
*** shashank_hegde has quit IRC18:10
*** lmdaly_ has quit IRC18:19
*** reedip has joined #openstack-kuryr18:21
*** garyloug has quit IRC18:22
*** reedip has quit IRC18:30
ivc_apuimedo updated, rebased and verified: https://review.openstack.org/#/c/404038/18:35
ivc_apuimedo also added https://review.openstack.org/#/c/407119/18:35
ivc_irenab, https://review.openstack.org/#/c/404038/ tested with both trusty and xenial (used local.conf tied to that specific patchset)18:36
ivc_irenab, local.conf: http://paste.openstack.org/show/591462/18:37
*** reedip has joined #openstack-kuryr18:43
*** shashank_hegde has joined #openstack-kuryr18:47
openstackgerritHongbin Lu proposed openstack/fuxi: Add devstack plugin for Fuxi  https://review.openstack.org/40224419:00
*** dougbtv has quit IRC19:08
*** dougbtv has joined #openstack-kuryr19:11
*** diogogmt has joined #openstack-kuryr19:15
*** dougbtv has quit IRC19:16
*** tonanhngo has joined #openstack-kuryr19:17
*** tonanhngo_ has joined #openstack-kuryr19:20
*** tonanhngo has quit IRC19:21
*** madgin has joined #openstack-kuryr19:22
*** tonanhngo_ has quit IRC19:25
*** mattmceuen has quit IRC19:28
*** tonanhngo has joined #openstack-kuryr19:31
*** tonanhngo has quit IRC19:36
*** dougbtv has joined #openstack-kuryr19:36
*** tonanhngo has joined #openstack-kuryr20:01
*** tonanhngo has quit IRC20:01
*** tonanhngo has joined #openstack-kuryr20:02
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork: Fix the check of docker installation  https://review.openstack.org/40717220:04
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork: Fix the check of docker installation  https://review.openstack.org/40717220:05
openstackgerritHongbin Lu proposed openstack/kuryr-kubernetes: Fix the check of docker installation  https://review.openstack.org/40717920:07
openstackgerritHongbin Lu proposed openstack/fuxi: Separate unit tests from fullstack tests  https://review.openstack.org/40393120:26
openstackgerritHongbin Lu proposed openstack/fuxi: Add basic fullstack volume tests  https://review.openstack.org/40394120:26
*** madgin is now known as mattmceuen20:40
*** oanson has quit IRC20:42
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittest with mock  https://review.openstack.org/40520321:30
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: Unittest with mock  https://review.openstack.org/40520321:37
openstackgerritDarla Ahlert proposed openstack/kuryr-libnetwork: [WIP]Unittest with mock  https://review.openstack.org/40520322:13
openstackgerritMarco Chiappero proposed openstack/kuryr-libnetwork: Fully support MAC address change on CreateEndpoint  https://review.openstack.org/40663622:30
openstackgerritMarco Chiappero proposed openstack/kuryr-libnetwork: [WIP] Driver based model for kuryr-libnetwork  https://review.openstack.org/40036522:37
*** pmannidi has joined #openstack-kuryr23:46

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!