Monday, 2016-04-18

*** salv-orlando has quit IRC00:06
*** outofmemory is now known as reedip00:17
*** banix has joined #openstack-kuryr02:08
*** banix has quit IRC02:08
*** tfukushima has joined #openstack-kuryr02:30
*** tfukushima has quit IRC02:33
openstackgerritMohammad Banikazemi proposed openstack/kuryr: Makes use of Neutron tags option  https://review.openstack.org/30662702:51
*** yuanying_ has quit IRC02:51
*** yamamoto_ has joined #openstack-kuryr02:59
*** yamamoto_ has quit IRC03:11
*** yamamoto_ has joined #openstack-kuryr03:22
*** yamamoto_ has quit IRC03:43
*** yuanying has joined #openstack-kuryr03:46
*** tfukushima has joined #openstack-kuryr03:52
*** tfukushima has quit IRC03:56
*** tfukushima has joined #openstack-kuryr04:07
*** salv-orlando has joined #openstack-kuryr04:09
*** salv-orlando has quit IRC04:13
*** yamamoto_ has joined #openstack-kuryr04:27
*** oanson has joined #openstack-kuryr04:29
*** tfukushima has quit IRC04:35
*** salv-orlando has joined #openstack-kuryr04:48
*** oanson has quit IRC04:50
*** oanson has joined #openstack-kuryr05:33
*** tfukushima has joined #openstack-kuryr05:36
*** tfukushima has quit IRC05:40
*** tfukushima has joined #openstack-kuryr05:45
*** huats_ has joined #openstack-kuryr07:21
*** lezbar has quit IRC07:39
*** lezbar has joined #openstack-kuryr08:14
*** yamamoto_ has quit IRC09:02
*** tfukushima has quit IRC09:02
*** yamamoto has joined #openstack-kuryr09:02
*** tfukushima has joined #openstack-kuryr09:05
*** tfukushima has quit IRC09:28
*** tfukushima has joined #openstack-kuryr09:28
*** salv-orl_ has joined #openstack-kuryr10:04
*** salv-orlando has quit IRC10:07
*** apuimedo has joined #openstack-kuryr11:28
*** tfukushima has quit IRC11:30
*** yamamoto has quit IRC12:01
*** tfukushima has joined #openstack-kuryr12:25
*** tfukushima has quit IRC12:34
*** yamamoto has joined #openstack-kuryr12:38
*** yamamoto has quit IRC12:45
*** yamamoto has joined #openstack-kuryr12:45
*** yamamoto has quit IRC12:50
*** yamamoto has joined #openstack-kuryr12:54
*** yamamoto has quit IRC12:56
*** yamamoto has joined #openstack-kuryr12:57
*** yamamoto has quit IRC13:02
*** yamamoto has joined #openstack-kuryr13:13
apuimedogsagie: ping13:14
gsagieapuimedo: pong13:18
irenabgsagie: apuimedo : did you run devstack recently?13:37
apuimedohuats_ did13:37
irenabI get issues when running stack.sh13:37
huats_I did indeed13:38
irenabubuntu?13:38
apuimedogsagie: google slides pptx importer is a piece of trash13:41
apuimedoit loses pictures and stuff. I'll convert it to odp first, see if it helps13:41
gsagiek13:41
irenabapuimedo: you just insist to use all the tools on linux :-)13:43
apuimedoirenab: I'm a freeman!13:44
irenabhuats_: did it run smoothly or you had to fix issues>13:44
gsagieapuimedo: i think if you upload ppt to google drive it converts it13:46
huats_irenab: I did on Ubuntu yep13:47
huats_I had one issue, not related to Devstack but kuryr... But i fixed it and it is on the repo  :)13:47
irenabhuats_: Ithanks, trying to rerun the stack.sh13:49
*** oanson has quit IRC13:55
apuimedogsagie: it does, and badly :P13:55
apuimedohuats_: irenab: can you do an fpaste of your local.conf for reference13:56
*** tfukushima has joined #openstack-kuryr13:56
irenabhttp://pastebin.com/qrxFscVG13:57
*** tfukushima has quit IRC13:57
irenabbut the issue I have is related to running docker:   stop: Unknown instance: 13:58
irenab2016-04-18 13:54:49.105 | Error on exit13:58
apuimedowhich ubuntu is this?13:58
irenabUbuntu 14.04.4 LTS14:01
huats_I used the same14:02
huats_irenab: why do you rerun the stack.sh14:02
irenabI think maybe I need to remove docker before rerunning the stack.sh14:02
huats_usually it is not a good idea IIRC14:02
huats_I think it is the case14:03
huats_(that you should remove docker)14:03
irenabtrying it, thanks14:04
huats_I'll have a look too14:05
huats_(but I have to say that I am clearly late to work on my talk for next week :))14:05
huats_(so I am trying to focus :))14:05
apuimedohuats_: link to the talk?14:06
huats_https://www.openstack.org/summit/austin-2016/summit-schedule/events/8307?goback=114:11
huats_apuimedo: not related to kuryr I agree :)14:11
apuimedocharging is always nice :-)14:11
huats_:)14:12
huats_irenab: I'll have a look if we need to clear docker in the devstack and I'll patch it if needed...14:12
irenabhuats_: I will let you know if it fixed the problem, it takes ages to deploy …14:13
huats_great !14:16
irenabhmm, got this error14:22
irenabdpkg: error processing package docker-engine (--configure):14:22
irenab2016-04-18 14:17:25.044 |  subprocess installed post-installation script returned error exit status 114:22
irenab2016-04-18 14:17:25.044 | Processing triggers for libc-bin (2.19-0ubuntu6.7) ...14:22
irenab2016-04-18 14:17:25.175 | Errors were encountered while processing:14:22
irenab2016-04-18 14:17:25.175 |  docker-engine14:22
irenab2016-04-18 14:17:25.803 | E: Sub-process /usr/bin/dpkg returned an error code (1)14:22
huats_ok14:25
huats_can you do14:25
huats_dpkg --remove --purge docker-engine14:25
huats_and then  stack.sh again ?14:25
huats_irenab: ^14:26
irenabdpkg: error: conflicting actions -P (--purge) and -r (--remove)14:26
huats_sorry :)14:27
huats_just use purge14:27
irenabrunning14:28
irenabit requests unstack first, so will take time ...14:28
*** tfukushima has joined #openstack-kuryr14:30
huats_ok14:33
huats_take your time :)14:33
irenabI wish it could be faster …14:41
irenabhuats_: well, back to where it started14:44
irenabdpkg: error processing package docker-engine (--configure):14:44
irenab2016-04-18 14:43:01.848 |  subprocess installed post-installation script returned error exit status 114:44
irenab2016-04-18 14:43:01.848 | Processing triggers for ureadahead (0.100.0-16) ...14:44
irenab2016-04-18 14:43:03.395 | Errors were encountered while processing:14:44
huats_:(14:44
irenab2016-04-18 14:43:03.395 |  docker-engine14:44
irenab2016-04-18 14:43:04.149 | E: Sub-process /usr/bin/dpkg returned an error code (1)14:44
irenab2016-04-18 14:43:04.158 | Error on exit14:44
huats_hum14:44
huats_let me check something (I have started a devstack too in the same time)14:44
irenabhuats_: need to go for a while, will sync later14:46
huats_ok14:47
openstackgerritMohammad Banikazemi proposed openstack/kuryr: Makes use of Neutron tags option  https://review.openstack.org/30662714:55
*** oanson has joined #openstack-kuryr15:04
*** oanson has quit IRC15:05
*** tfukushima has quit IRC15:11
*** asti has joined #openstack-kuryr15:13
*** salv-orlando has joined #openstack-kuryr16:04
*** salv-orl_ has quit IRC16:06
apuimedogsagie: new slides sent16:20
apuimedogotta go pack the luggage16:20
irenabhuats_: I got new clean VM and devstack completed fine16:38
huats_ok16:38
huats_!16:38
huats_great16:38
irenabseems Jaume had similar issue:  https://review.openstack.org/#/c/290702/16:41
*** banix has joined #openstack-kuryr17:03
apuimedo;-)18:00
apuimedoI think I'm going to hijack jaume's patch to get it in :P18:00
*** salv-orlando has quit IRC18:13
*** salv-orlando has joined #openstack-kuryr18:13
*** salv-orlando has quit IRC18:18
*** thingee has joined #openstack-kuryr18:22
thingeegsagie: Kuryr's summit schedule shows TBD for a fishbowl session. Can we have that updated today please?18:22
*** salv-orlando has joined #openstack-kuryr18:24
openstackgerritKyle Mestery proposed openstack/kuryr: README.md: Cleanup documentation  https://review.openstack.org/30742019:06
openstackgerritKyle Mestery proposed openstack/kuryr: README.md: Cleanup documentation  https://review.openstack.org/30742019:15
*** yamamoto has quit IRC19:39
*** salv-orlando has quit IRC19:47
*** yamamoto has joined #openstack-kuryr19:54
*** salv-orlando has joined #openstack-kuryr20:33
mesteryI'm seeing issues with Kuryr using Vagrant: http://paste.openstack.org/show/494512/20:35
mesterygsagie banix apuimedo: Any ideas on what's happening? ^^^20:36
banixmestery: hmmmm not sure what’s the problem20:39
mesterybanix: Bummer. Looks like it's not loading the kuryr driver in docker perhaps?20:40
banixmestery: yes looks like the error happens when trying top load tge plugin20:49
banixis the kuryr.json ther:20:52
banixsomething like this:$ cat /usr/lib/docker/plugins/kuryr/kuryr.json20:52
banix{20:52
banix    "Name": "kuryr",20:52
banix    "Addr": "http://127.0.0.1:2377"20:52
banix}20:52
* mestery looks20:54
mesteryYup20:55
mesteryHas the same thing as you have20:55
mesteryAnd my kuryr is running20:55
mesteryvagrant@devstack:~$ ps axuw|grep kuryr20:55
mesteryroot      9564  0.0  0.0  62096  2028 pts/7    S+   20:27   0:00 sudo PYTHONPATH=:/opt/stack/kuryr SERVICE_USER=admin SERVICE_PASSWORD=pass SERVICE_TENANT_NAME=admin SERVICE_TOKEN=pass IDENTITY_URL=http://127.0.0.1:5000/v2.0 python /opt/stack/kuryr/scripts/run_server.py --config-file /etc/kuryr/kuryr.conf20:55
mesteryroot      9565  0.0  0.9  99336 38240 pts/7    S+   20:27   0:01 python /opt/stack/kuryr/scripts/run_server.py --config-file /etc/kuryr/kuryr.conf20:55
mesteryvagrant  17882  0.0  0.0  10464   920 pts/24   S+   20:55   0:00 grep kuryr20:55
mesteryvagrant@devstack:~$20:55
*** fawadkhaliq has joined #openstack-kuryr20:57
apuimedomestery: ?20:58
mesteryapuimedo: Kuryr no worky for me20:58
apuimedolatest devstack?20:58
apuimedodocker 1.10?20:58
apuimedoovs/ovn?20:58
mesteryYes20:59
mesteryLatest of everything20:59
apuimedommm.... Irena got it working today and huats_ did use it last week21:00
apuimedonot with vagrant though21:00
mesteryapuimedo: It's an odd error, did you see it? Almsot like Docker can't talk to Kuryr21:01
apuimedomestery: may it be that docker started before kuryr?21:01
mesteryapuimedo: Let me check21:01
mesteryapuimedo: So, in the screen session, the docker tab is indeed before the kuryr tab21:01
mesteryBut Vagrant just runs a normal devstack21:02
mesterySo unsure why this would be different21:02
apuimedomestery: try stopping docker and starting it again21:02
* mestery buckles in and tries21:02
apuimedoprobably the registration got botched21:02
mesteryapuimedo: same21:02
apuimedo(╯° °)╯彡┻━┻21:03
apuimedocan you telnet to 127.0.0.1 2377 ?21:03
* mestery tries21:04
mesteryYup21:04
mesteryworks like a champ21:04
apuimedook, so at least kuryr is listening21:04
mesteryYup21:04
apuimedomestery: the error is the same then? Still trying to reach 2375? instead of 2377?21:04
mesteryYeah21:05
apuimedommm21:06
apuimedomestery: can you try changing the docker engine to listen on the unix socket?21:06
mesteryapuimedo: OK, let me try that21:07
banixhave to run but will be on line later tonight21:07
*** banix has quit IRC21:07
apuimedofawadkhaliq: I'll update you on the slides tomorrow night, gotta travel21:08
apuimedosorry about the delay :(21:08
fawadkhaliqapuimedo: thanks thanks!21:08
apuimedomestery: I assume you have that running on a vagrant on your laptop, right?21:09
mesteryapuimedo: Yes21:09
mesteryand with a change to local socket I see this:21:09
mesteryvagrant@devstack:~$ sudo docker -H unix:///var/run/docker.sock network create -d kuryr foo21:09
mesteryAn error occurred trying to connect: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.23/networks/create: EOF21:09
mesteryvagrant@devstack:~$21:09
apuimedomestery: do the other docker commands work?21:10
mesteryapuimedo: Ack21:10
mesteryThey did when it was listening on the socket as well21:10
apuimedomestery: very well21:11
mesteryHeh :)21:11
apuimedoor very wrong :P21:11
mesteryYeah, that one! :)21:11
apuimedomestery: exact docker version?21:11
mesteryvagrant@devstack:~$ docker --version21:11
mesteryDocker version 1.11.0, build 4dc599021:11
mesteryvagrant@devstack:~$21:11
apuimedo1.11... That looks scary... Knowing the lack of bc regard these folks have21:12
mesteryapuimedo; lol21:12
mesteryWell, that's what devstack pulls down21:13
mesteryAnd yeah, my guess is now that docker has screwed us again21:13
mesteryBecause: Backwards compatability! Why does it matter?21:13
apuimedobut I think Irena got the devstack working today21:13
apuimedolet's do a quick check21:14
mesteryThanks apuimedo :)21:14
apuimedomestery: ubuntu, right?21:14
mesteryapuimedo: Ack21:14
apuimedomestery: trusty?21:14
mesteryack21:15
apuimedohttps://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.10.3-0~trusty_amd64.deb21:15
apuimedotry this one21:15
apuimedoif it works. I'll start a jihad21:15
* mestery downloads21:16
mesteryThat did it21:18
mesteryIt worked21:18
mesterySo21:18
mesterydocker people screwed us :(21:18
mesteryMeh21:18
mesteryThey do this all the time!21:18
mesteryEvery 3 months I try to use kuryr, docker has broken backwards compatibility again21:18
apuimedomestery: you are the harbinger of bad bc breakages21:19
mesteryapuimedo: Indeed I am21:19
* mestery shakes his fist at docker ... again21:19
apuimedoI really can't believe the sloppiness with bc21:20
apuimedoI should have worked on nova. This doesn't happen with libvirt21:21
mesteryrofl21:21
mesteryI know right?21:21
mesteryI mean21:21
mesteryMan21:21
mesteryI don't think Docker really cares about anything they make pluggable21:21
apuimedommm... I'll fire up some big VM to try to debug that21:21
mesteryThanks apuimedo :)21:21
apuimedodocker has problems of care21:22
mesteryapuimedo: +100021:22
apuimedoI didn't find any change in the release notes that would signify such breakage21:22
mesteryOf course you didn't :)21:22
apuimedomestery: checking anything specific?21:25
mesteryapuimedo: MEaning?21:25
apuimedoif you are running kuryr with some specific goal21:25
mesteryAh, well, for a demo, so yeah :)21:25
apuimedowhat will you show?21:26
mesteryapuimedo: Really basic stuff, it's for an IBM sponsored talk in Austin next week that Phil Estes and I are giving21:30
apuimedoah, cool21:31
apuimedomestery: since I'm starting devstack, maybe I'll test and get merged the container based kuryr21:31
mesteryapuimedo: Coolio!21:32
apuimedoI already have it for the kubernetes integration21:32
apuimedoautomated builds of the api watcher21:32
mesteryCool21:33
apuimedothe problem with libnetwork is the stupidity of docker needing to be restarted when you add plugins21:33
mesterylol21:33
apuimedoand that you can't have a plugin register itself by calling the docker API21:33
apuimedowhich makes docker based plugins needlessly painful to deploy21:34
apuimedoone would imagine they want plugins to be container based21:35
mesteryapuimedo: Well, one could imagine they really don't even want plugins21:35
mesteryAnd that plugins were just to get them some mindshare21:35
mesteryBecause they are building everything themselves :)21:35
apuimedomestery: indeed21:35
apuimedosocketplane...21:35
mesteryRight21:36
apuimedoit's quite a difference with how the kubernetes network group works21:36
mesteryYeah21:36
*** asti has quit IRC21:49
salv-orlandolet's ditch libnetwork and do CNI!!! Hasta la revoluccion :)21:52
apuimedosalv-orlando: you are tempting me :P21:54
apuimedosalv-orlando: have you tried standalone rkt cni?21:55
salv-orlandoapuimedo: not yet, but I did not want to bring my revolution as far as ditching docker as well ;)21:55
apuimedosalv-orlando: well, with docker it is very cumbersome to use cni21:56
salv-orlandoon the other hand I also know little about CNI behaviour wrt bw compatibility21:56
apuimedoyou basically have to create a docker container, then use CNI on it21:56
apuimedothen you can start the real container that does things21:56
apuimedoit really sucks21:57
apuimedoand I wish kubernetes didn't have to do that21:57
apuimedo(and hope in rkt it is not the case, but I haven't checked, not to be disappointed)21:57
salv-orlandoindeed. Considering that CNI is native in rkt I expect a seamless experience21:58
salv-orlandonevertheless there's no revolution as it's not up to kuryr to decide whether docker should run CNI or CNM21:58
apuimedosalv-orlando: :'(21:58
salv-orlandoso, I guess the only possible thing to do is...21:58
salv-orlandosuck it up!21:59
apuimedosometimes, specially when docker bc breaks21:59
apuimedoI wish everybody would just kick docker out and take over runc21:59
apuimedoand do sane things21:59
*** fawadkhaliq has quit IRC22:00
mesteryYeah22:00
mesteryno more docker!22:00
apuimedogive karma time22:02
salv-orlandolet22:02
salv-orlandos go back to UML22:02
apuimedosalv-orlando: you are risking a banning from this channel22:02
apuimedoUML and design patterns are haram22:02
apuimedocontainers based on ubuntu images when they don't need a full system is also22:03
apuimedoI should compile my religion tenets22:03
salv-orlandoapuimedo: but I meant user mode linux!!!22:03
salv-orlandothe grandfather of all container technology22:04
apuimedosalv-orlando: ok then. You almost worried me about the kuryr architecture session22:04
salv-orlandowell that's chroot actually22:04
apuimedoif I see uml in a code review or working session I may get an aneurism22:04
apuimedosalv-orlando: have you taken a look at lxd?22:05
salv-orlandoapuimedo: yes, when it was announced22:05
salv-orlandoI was not impressed back then, but I have not followed developments22:05
apuimedowell, apart from the decision to base it on zfs, which is against my religion too22:06
apuimedoit feels like it could have been done in libvirt22:06
apuimedosalv-orlando: how's it going with the policy?22:07
apuimedoso far I'm thinking just making policy object -> security group22:07
apuimedoand then add/remove SGs from the ports22:07
salv-orlandoapuimedo: today a blog post on the API for policy has been published here: http://blog.kubernetes.io/2016/04/Kubernetes-Network-Policy-APIs.html22:08
salv-orlandoI'm working into adding some reliable caches to my watcher22:08
apuimedosalv-orlando: for which objects?22:08
salv-orlandonamespaces, policys and pods22:09
apuimedoI made a very naive cache for the reconnects only22:09
apuimedoeach watcher gets a lru cache22:09
salv-orlandomy cache is probably even more naive than that22:09
salv-orlandoit's a dict ;)22:09
apuimedo(in raven each endpoint gets a watcher)22:09
apuimedoI inherit from ordereddict to do the eviction22:09
salv-orlandoanyway at some point I will consider whether use Raven too for direct OVN integration22:09
apuimedosalv-orlando: lazy minds think alike22:09
apuimedosalv-orlando: there is one design decision I have to take this week22:10
apuimedothat I'm not sure about yet22:10
salv-orlandoapuimedo: as for your thinking yes a k8s has been modeled in a way that's very similar to the security group22:10
salv-orlandowith the different in the "from" bits22:10
salv-orlandothat in openstack is either a cidr or another security group22:10
salv-orlandowhile in kubernetes is a pod selector22:11
apuimedoI'm thinking on having the namespace watcher spawn watchers for the other entities for each new namespace22:11
salv-orlandothat is quite unlikely to map to a cidr22:11
apuimedoand cancel the tasks when the namespace is deleted22:11
apuimedoinstead of having a watcher per resource22:11
apuimedothat takes care of all the namespaces22:11
salv-orlandoapuimedo: I've done something similar, perhaps. As the policys are only namespaced everytime a namespace event occurs a policy watcher is started22:12
apuimedosalv-orlando: that "from" part sucks22:12
apuimedosalv-orlando: sounds similar22:12
salv-orlandoapuimedo: It makes sense from the k8s consumer point of view though.22:12
apuimedomy idea was that if I do it with multiple resource watchers, I can potentially have them scale out22:12
salv-orlandoMapping into Neutron, OVN, or whatever uses IP addresses is a bit of a pain22:12
apuimedoin multiple processes/machines22:12
apuimedosalv-orlando: well. When you make something easily usable for people that don't know what they are doing, you have to pay the price in hair loss22:13
salv-orlandoapuimedo: that is an idea. From my perspective however in order to think about "scaling out" I need first to understand what the scale bottlenecks are22:13
apuimedoI now pull my beard, it is more resilient22:13
salv-orlandoin my experience reality is often different from modeling ;)22:14
apuimedosalv-orlando: of course, not going to write any multi process anytime soon22:14
apuimedobut having watchers small and independent sounds nice in my head22:14
salv-orlandoapuimedo: for instance, just as an example, we do not know what kind of load a watcher per resource will ultimately put on etcd22:15
apuimedoI hear the echo22:15
salv-orlandoyou might scale out raven but kill etcd ;)22:15
apuimedosalv-orlando: the API, not etcd22:15
apuimedoright?22:15
apuimedowhich is worse, because etcd you could just put a ton of them22:15
apuimedoI'm not sure it is as easy, putting a lot of api endpoints22:16
apuimedosalv-orlando: I'm probably repeating myself but... What would be so wrong if the extensibility was built as a scheduler plugin?22:17
apuimedoit would be executed right in the api controllers22:17
apuimedoless moving pieces22:18
apuimedono back and forth from cni -> api22:18
apuimedoand if you namespaced the attributes they can add, you could run multiple of them at the same time22:18
apuimedoI'm probably missing something very evident, but I have not found it yet22:19
apuimedomestery: ok, devstack just finished22:34
apuimedoand yeah, it reproduced nicely22:34
mesterygood!22:40
apuimedomestery: it looks like a big libnetwork bug. I suspect they may have broken the non-socket plugin usage22:57
apuimedo*non-unix-socket22:58
*** asti has joined #openstack-kuryr23:34
*** salv-orlando has quit IRC23:35
mesteryapuimedo: Bummer!23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!