| gmmaha | sdake: vhosakot: i tried finding the files again with a re-deploy but nothing | 00:00 |
|---|---|---|
| gmmaha | i believe tonight i will re-isntall the base OS on the compute hosts and then start a fresh deploy | 00:00 |
| gmmaha | hopefully things might turn out different tomorrow | 00:00 |
| gmmaha | thanks for helping me along with this.. | 00:00 |
| sdake | gmmaha it is possible ubuntu changed its binary install of libvirt | 00:02 |
| sdake | you have a newer version then I have | 00:02 |
| sdake | it is possible this binary version is broken | 00:02 |
| sdake | but the gate is still working for ubuntu so that is unlikely | 00:02 |
| gmmaha | sdake: very very possible | 00:02 |
| sdake | so, don't know what to tellyou ;( | 00:02 |
| gmmaha | sdake: aaah.. gate pulls the upstream image i guess | 00:02 |
| sdake | gate builds the image | 00:02 |
| gmmaha | aah is that the default setting or do i have to enable a diff config for that | 00:03 |
| gmmaha | maybe if i can re-create the gate settings, i will avoid a possible trip | 00:03 |
| vhosakot | gmmaha: frankly, no issue in kolla should need a re-image of the OS on the host node... I'd try like upgrade/downgrade libvirt, get latest kolla master, upgrade images, dig libvirt logs, etc etc | 00:05 |
| sbezverk | sdake how do you force path to be rechecked by jenkins? | 00:05 |
| sbezverk | sdake /path/patch/ | 00:06 |
| sdake | you type "recheck" | 00:06 |
| sdake | as the onl line in the review when clicking reply | 00:06 |
| sbezverk | sdake hm, I did that nothing happened.. | 00:06 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Test to see if this fixes the gate https://review.openstack.org/298460 | 00:07 |
| sdake | it takes awhile for anything toh pappen sbezverk | 00:07 |
| sdake | if you want to see results, check out zuul.openstack.org | 00:07 |
| sdake | its the masster job runner for openstack | 00:07 |
| vhosakot | sbezverk: I've seen recehck take a couple of mins to start... depends on ZUUL queue status I think | 00:07 |
| sdake | you know, from ghostbusters, the gatekeeper and all that | 00:07 |
| vhosakot | gmmaha: you still have interest to debug more ? | 00:08 |
| vhosakot | gmmaha: one thing i'd check is if your /run mount is shared on host | 00:08 |
| vhosakot | libvirt's socket is in /run/libvirt/ | 00:09 |
| vhosakot | mount --make-shared /run | 00:09 |
| sdake | well rax.ord got tkane out of the gate | 00:18 |
| sdake | \o/ ;) | 00:18 |
| vhosakot | tkane ? | 00:18 |
| sdake | taken | 00:19 |
| sdake | removed | 00:19 |
| vhosakot | the one with issues right ? | 00:19 |
| vhosakot | the one with two interfaces ? | 00:19 |
| *** salv-orlando has joined #kolla | 00:24 | |
| *** ssurana has quit IRC | 00:27 | |
| *** weiyu_ has quit IRC | 00:28 | |
| *** salv-orlando has quit IRC | 00:28 | |
| sdake | thats the one | 00:30 |
| sdake | asalkeld pingola | 00:33 |
| *** ccesario_ has joined #kolla | 00:33 | |
| sdake | https://review.openstack.org/#/c/298460/2 | 00:33 |
| patchbot | sdake: patch 298460 - kolla - [WIP] Test to see if this fixes the gate | 00:33 |
| asalkeld_ | hi sdake | 00:35 |
| *** sdake_ has joined #kolla | 00:35 | |
| *** sdake has quit IRC | 00:38 | |
| *** jtriley has joined #kolla | 00:38 | |
| *** achanda has joined #kolla | 00:41 | |
| *** ccesario__ has joined #kolla | 00:45 | |
| *** ccesario_ has quit IRC | 00:45 | |
| *** achanda has quit IRC | 00:46 | |
| *** sdake_ has quit IRC | 00:47 | |
| *** sdake has joined #kolla | 00:47 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "Fix consistently failing CentOS gate" https://review.openstack.org/297084 | 00:54 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] try removing setuptools and pytz https://review.openstack.org/298479 | 00:54 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] try removing setuptools and pytz https://review.openstack.org/298479 | 00:55 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "Fix consistently failing CentOS gate" https://review.openstack.org/297084 | 00:55 |
| *** achanda has joined #kolla | 01:01 | |
| dmsimard | jinja2 is required by ansible too | 01:04 |
| *** rhallisey has quit IRC | 01:05 | |
| dmsimard | sdake: ^ not just zookeeper | 01:05 |
| *** Liuqing has joined #kolla | 01:06 | |
| sdake | dmsimard jinja2 gets instlled on he host nodes by ansible by pulling it in | 01:09 |
| *** weiyu has joined #kolla | 01:12 | |
| *** pbourke-home has joined #kolla | 01:12 | |
| *** pbourke-home has quit IRC | 01:14 | |
| *** Liuqing has quit IRC | 01:15 | |
| *** Marga_ has quit IRC | 01:16 | |
| *** banix has quit IRC | 01:17 | |
| sdake | dmsimard in th econtainer itself jinja2 is apparnetly needed by zookeeper in some way | 01:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] try removing setuptools and pytz https://review.openstack.org/298479 | 01:23 |
| vhosakot | sdake: to Matt's reply you should have also added "Yes, kolla does replace DevStack with OpenStack in containers. In fact, we already do that in the kolla gate today. Kolla's gate is muct closer to a production cloud as it does not use DevStack to bringup/test OpenStack" :) | 01:29 |
| sdake | enough popcorn forone day ;) | 01:30 |
| vhosakot | haha | 01:30 |
| sdake | although on openstack-release, it was suggested we had co-gating jobs to the compute-kit ;) | 01:30 |
| vhosakot | I think kolla gate is closer to production than DevStack gate .. | 01:30 |
| sdake | had/add | 01:30 |
| sdake | vhosakot trust me i was tempted | 01:31 |
| vhosakot | to use/add DevStack ? | 01:31 |
| sdake | to make that line of argument | 01:31 |
| sdake | but some fights are better left unfought | 01:31 |
| vhosakot | ah ok | 01:33 |
| *** allen_gao has quit IRC | 01:42 | |
| *** allen_gao has joined #kolla | 01:46 | |
| *** weiyu has quit IRC | 01:51 | |
| *** Liuqing has joined #kolla | 01:51 | |
| *** banix has joined #kolla | 01:54 | |
| *** unicell has quit IRC | 01:56 | |
| *** Marga_ has joined #kolla | 01:56 | |
| *** ccesario__ has quit IRC | 01:57 | |
| *** ccesario__ has joined #kolla | 01:57 | |
| *** tfukushima has joined #kolla | 01:57 | |
| *** jtriley has quit IRC | 01:57 | |
| *** Marga_ has quit IRC | 02:01 | |
| *** weiyu_ has joined #kolla | 02:03 | |
| *** jtriley has joined #kolla | 02:17 | |
| *** Marga_ has joined #kolla | 02:17 | |
| *** Marga_ has quit IRC | 02:21 | |
| vhosakot | does kolla work if network_interface: "enp2s0f1" in globals.yml instead of eth0 or eth1 ? | 02:30 |
| *** allen_gao has quit IRC | 02:32 | |
| sdake | vhosakot any idea what is wrong with https://jenkins04.openstack.org/job/gate-kolla-dsvm-build-ubuntu-source/544/console | 02:35 |
| sdake | vhosakot i have some variant of that on my gear and it does work | 02:36 |
| sdake | i think mine is enp1s0f1 or something | 02:36 |
| sdake | that is an intel 10gige card | 02:36 |
| vhosakot | sdake: ah ok... | 02:36 |
| vhosakot | I dont see any error in that jenkins link | 02:37 |
| sdake | rax.ord is out of service - \o/ ;) | 02:39 |
| vhosakot | sdake: I see it is stuck at "ansible-playbook -i /tmp/kolla/raw_inventory tools/setup_nodes.yml" which should take 1-2 mins max.... | 02:40 |
| sdake | yes i know | 02:40 |
| vhosakot | sdake: it should timeout eventually | 02:40 |
| sdake | first time i'e see nthat happen | 02:41 |
| vhosakot | is this rax-ord ? | 02:41 |
| *** allen_gao has joined #kolla | 02:41 | |
| sdake | no idea | 02:41 |
| sdake | vexxhost-ca | 02:41 |
| *** weiyu_ has quit IRC | 02:42 | |
| *** weiyu has joined #kolla | 02:42 | |
| vhosakot | not even the fist task is seen.. so, looks like SSH issue to ansible host... | 02:44 |
| sdake | 2016-03-29 01:29:13.786 | + ssh-keygen -f /home/jenkins/.ssh/kolla -N '' | 02:44 |
| sdake | maybe we need special loe for vexxhost | 02:45 |
| sdake | loe/love | 02:45 |
| vhosakot | I meant the first taks of "ansible-playbook -i /tmp/kolla/raw_inventory tools/setup_nodes.yml" is not printed | 02:46 |
| vhosakot | should we add -T 300 to "ansible-playbook -i /tmp/kolla/raw_inventory tools/setup_nodes.yml" so it times out after 5 mins in gate | 02:47 |
| vhosakot | I see ansible-playbook has a -T flag to force timeout http://linux.die.net/man/1/ansible-playbook... if this makes sense, I will submit aPS | 02:47 |
| sdake | if it times out i want it to fail the gate immediately | 03:01 |
| sdake | but ya that sounds good | 03:01 |
| sdake | clarkb said jenkins may have been interruptedd and not noticed | 03:01 |
| gmmaha | vhosakot: i thought the /run mount shared was purely just for the controllers | 03:02 |
| gmmaha | does that need to be done for compute hosts as well? | 03:02 |
| gmmaha | sdake: agreed.. thats an extreme measure to re-image the metals.. :) | 03:03 |
| vhosakot | no... on all hosts... | 03:03 |
| vhosakot | gmmaha: nova_libvirt mounts /run as well..... https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/start_compute.yml#L13 | 03:03 |
| *** ccesario__ has quit IRC | 03:03 | |
| *** klint has joined #kolla | 03:03 | |
| gmmaha | vhosakot: aaah | 03:04 |
| gmmaha | we have been making just the controllers as shared mount | 03:04 |
| ntpttr | gmmaha: we did have it on all hosts, it wouldn't pass the prechecks otherwise | 03:04 |
| ntpttr | gmmaha: I set it for them all | 03:04 |
| gmmaha | ntpttr: aah | 03:04 |
| gmmaha | ok.. thanks.. :) | 03:04 |
| vhosakot | it will be lost every time you reboo the node | 03:04 |
| gmmaha | didnt know.. | 03:04 |
| ntpttr | vhosakot: that I didn't know, I was wondering why we lost it every once and a while | 03:05 |
| *** ccesario_ has joined #kolla | 03:05 | |
| *** SiRiuS_ has quit IRC | 03:05 | |
| *** banix has quit IRC | 03:09 | |
| gmmaha | vhosakot: any chance i can find the libvirt version on the container image? | 03:09 |
| gmmaha | guess when its in the restarting state, i cant do much | 03:09 |
| vhosakot | mount | grep run but it wont show if it is hsared | 03:12 |
| vhosakot | shared | 03:12 |
| vhosakot | just show if /run is a mountpont ir not | 03:12 |
| gmmaha | vhosakot: it is mounted on all the machines | 03:13 |
| sdake | gmmaha you can | 03:13 |
| vhosakot | gmmaha: your nova_libvirt container is not even running to check the version of libvirt in it.... I do this | 03:14 |
| sdake | docker exec -it contaienr_name:2.0.0 bash | 03:14 |
| sdake | then use dpkg -l | grep libvirt | 03:14 |
| vhosakot | $ docker exec nova_libvirt /usr/sbin/libvirtd --version | 03:14 |
| vhosakot | /usr/sbin/libvirtd (libvirt) 1.3.1 | 03:14 |
| vhosakot | my container is running libvirt 1.3.1 | 03:14 |
| sdake | gmmaha docker exec to docke run above | 03:14 |
| sdake | that will ru nthe container in a new container instance | 03:14 |
| sdake | and allow you to inspect the contents | 03:15 |
| *** jtriley has quit IRC | 03:15 | |
| vhosakot | yep... dpkg also shows 1.3.1-ubuntu for me | 03:15 |
| *** weiyu has quit IRC | 03:15 | |
| *** weiyu has joined #kolla | 03:17 | |
| gmmaha | sdake: vhosakot: i am not able to exec into the nova_libvirt container. get the error Error response from daemon: Container nova_libvirt is restarting, wait until the container is running | 03:18 |
| gmmaha | when you said docker exec container_name:2.0.0 bash | 03:18 |
| gmmaha | i am guessing here the continaer name is either kollaglue/source-ubuntu..... or the registry path of the container? | 03:19 |
| sdake | kollaglue/source-ubuntu-nova-libvirt:2.0.0 I htink | 03:19 |
| *** Marga_ has joined #kolla | 03:19 | |
| sdake | docker run | 03:19 |
| sdake | not exec | 03:19 |
| gmmaha | aaah | 03:20 |
| gmmaha | let me try.. thanks | 03:20 |
| gmmaha | sdake: i am running 1.3.1 | 03:21 |
| gmmaha | /usr/sbin/libvirtd --version | 03:22 |
| gmmaha | /usr/sbin/libvirtd --version | 03:22 |
| gmmaha | /usr/sbin/libvirtd (libvirt) 1.3.1 | 03:22 |
| *** Marga_ has quit IRC | 03:24 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 03:24 |
| *** vhosakot has quit IRC | 03:24 | |
| *** achanda has quit IRC | 03:38 | |
| *** Marga_ has joined #kolla | 03:39 | |
| *** Marga__ has joined #kolla | 03:44 | |
| *** ayoung has quit IRC | 03:46 | |
| *** Marga_ has quit IRC | 03:47 | |
| *** achanda has joined #kolla | 03:48 | |
| *** ccesario_ has quit IRC | 03:49 | |
| sdake | gmmaha apparently taht is what vhosakot has as well | 03:55 |
| gmmaha | sdake: yup.. | 03:55 |
| gmmaha | i am cleaned all the containers, removed al the images, mvoed /usr/local/share/kolla elsewhere (cause now i am purely paranoid) | 03:55 |
| gmmaha | and started a clean build | 03:55 |
| gmmaha | fingers crossed.. | 03:55 |
| gmmaha | :| | 03:55 |
| sdake | hmm | 03:57 |
| sdake | pip show kolla | 03:57 |
| sdake | gmmaha | 03:57 |
| *** alisonh has joined #kolla | 03:57 | |
| gmmaha | sdake: http://paste.openstack.org/show/492184/ | 03:57 |
| sdake | gmmaha unlike the directions recommend if your doing development (which you are) you dont want to use pip install kolla | 03:57 |
| gmmaha | though all the commands i will be using now will be from within the kolla source dir | 03:58 |
| sdake | use pip remove kolla | 03:58 |
| sdake | or uninstall kolla | 03:58 |
| gmmaha | sdake: yup. will do | 03:58 |
| sdake | gmmaha i a mmo reintereted in the libvirt vereion you have in your container though - the package version not libvirt version | 03:59 |
| sdake | dpkg -l libvirtd | 03:59 |
| gmmaha | sdake: just blew away all the containers | 04:00 |
| gmmaha | damn, i got it, but didnt get it copied out :( | 04:00 |
| gmmaha | and i am not able to get tmux to scroll back that far after i ran a build on top of it | 04:01 |
| ntpttr | gmmaha: ctrl b + ] doesn't work? | 04:03 |
| gmmaha | ntpttr: doesnt scroll that far up :9 | 04:04 |
| gmmaha | :( | 04:04 |
| ntpttr | gmmaha: that is the worst | 04:04 |
| gmmaha | only 2000 line scroll back | 04:04 |
| gmmaha | maybe we should increase the scroll back to 50,000 | 04:04 |
| *** achanda has quit IRC | 04:04 | |
| gmmaha | sdake: when we build kolla from source nad deploy frm there, where do all the Dockerfiles get stored? | 04:08 |
| gmmaha | sdake: hitting haproxy error. http://paste.openstack.org/show/492186/ | 04:09 |
| *** alisonh has left #kolla | 04:09 | |
| sdake | gmmaha there is a log in the topic | 04:09 |
| sdake | when you build from git the dockerfile are stored in your git repo | 04:10 |
| sdake | if ou do a pip install on them, they get installed in /usr/include/z | 04:10 |
| gmmaha | sdake: aah | 04:10 |
| sdake | but if you do a pip install to start with, the wrong dockerfiles (the ones your not editing) get installed | 04:10 |
| sdake | and kolla build places priority on those instead of the git repo | 04:11 |
| sdake | it should place prioority on the git repo files first | 04:11 |
| gmmaha | sdake: aaah | 04:11 |
| gmmaha | so best to remove them all and then try the install | 04:11 |
| gmmaha | ok.. let me try that out | 04:11 |
| sdake | pip uninstlal kolla will do the trick | 04:11 |
| sdake | asalkeld you about | 04:12 |
| *** pbourke-home has joined #kolla | 04:15 | |
| *** salv-orlando has joined #kolla | 04:16 | |
| *** pbourke-home has quit IRC | 04:17 | |
| *** weiyu has quit IRC | 04:19 | |
| gmmaha | sdake: did that and now nothing has been installed from the package and is used.. | 04:23 |
| gmmaha | hit this error.. http://paste.openstack.org/show/492188/ | 04:23 |
| sdake | an improvement | 04:24 |
| sdake | whta is 127.0.1.1? | 04:24 |
| gmmaha | sdake: hopefully!! :) | 04:24 |
| gmmaha | guessing thats just the local machine | 04:24 |
| sdake | remove it from /etc/hosts | 04:25 |
| gmmaha | aah | 04:25 |
| sdake | jfz11r02h03 should resole to an ip address | 04:25 |
| sdake | that is the interface's main address | 04:25 |
| sdake | this is why libvirt was busted i suspect | 04:25 |
| sdake | but who knows | 04:25 |
| sdake | lets debug one thing at a tem | 04:25 |
| gmmaha | sdake: agreed.. flushing it on all the machines now | 04:26 |
| sdake | manjeets had that on all of his hosts as well | 04:26 |
| sdake | and it made no sense to me why it was there | 04:26 |
| sdake | it wasn't in his ip netns list | 04:26 |
| sdake | rather ip link show | 04:26 |
| gmmaha | sdake: i am guessing either cobbler/bifrost install does it | 04:26 |
| *** salv-orlando has quit IRC | 04:28 | |
| gmmaha | sdake: did it all and the libvirt container is still on the reboot cycle :( | 04:45 |
| gmmaha | sdake: these are the packages that i have | 04:46 |
| gmmaha | ii libvirt-bin 1.3.1-1ubuntu4~cloud0 amd64 programs for the libvirt library | 04:46 |
| gmmaha | ii libvirt0:amd64 1.3.1-1ubuntu4~cloud0 amd64 library for interfacing with different virtualization systems | 04:46 |
| sdake | my next recommendatoin is complicated | 04:48 |
| gmmaha | sdake: fire away | 04:48 |
| sdake | do docker run libvirt_container:2.0.0 bash | 04:49 |
| sdake | try libviert --listen | 04:49 |
| sdake | libvirt --listen | 04:49 |
| sdake | if that doen't work | 04:49 |
| sdake | use apt-get to install strace | 04:49 |
| gmmaha | sdake: http://paste.openstack.org/show/492191/ | 04:50 |
| sdake | try just ilbvirtd | 04:50 |
| sdake | you notice how libvirt is running and not complaining about the permissions o th file | 04:51 |
| gmmaha | sdake: libvirtd isnt starting cause of the certificate | 04:51 |
| sdake | figure out the confg option in /etc/libvirt | 04:52 |
| sdake | to turn that off | 04:52 |
| sdake | i dont know what it is | 04:52 |
| sdake | google can help here | 04:52 |
| gmmaha | sdake: got it | 04:52 |
| gmmaha | sdake: for it to work.. seems like libvirtd is running | 05:08 |
| gmmaha | but with these error messages http://paste.openstack.org/show/492192/ | 05:09 |
| sdake | I think libvirt_admin is being configured by kolla in some way | 05:11 |
| sdake | and its probably pointing to the wrong location | 05:11 |
| sdake | can you run that by editing gthe json and run it through strace | 05:11 |
| *** achanda has joined #kolla | 05:12 | |
| sdake | you will need to use strace in the binary | 05:12 |
| sdake | in the dockerife | 05:12 |
| sdake | by installling it | 05:12 |
| sdake | th ereaon we want strace output is to see what libvirt is doing to read the admin so file | 05:13 |
| gmmaha | sdake: for sure | 05:14 |
| gmmaha | i have the docker container running | 05:14 |
| gmmaha | install strace and run libvirtd through it? | 05:15 |
| gmmaha | sdake: aah now i understood.. | 05:16 |
| gmmaha | fixing it and will try it out | 05:16 |
| sdake | do a deployment | 05:16 |
| gmmaha | will fix the dockerfile and the json | 05:16 |
| sdake | with strae in the jinja2 template | 05:16 |
| * gmmaha is now very close to braindead | 05:17 | |
| *** tfukushima has quit IRC | 05:18 | |
| sdake | https://github.com/pypa/pip/issues/1117 | 05:21 |
| gmmaha | sdake: i am guessing when i do build again in the poroject ansible will only rebuild the docker containers if there is a change in them right? | 05:24 |
| sdake | gmmaha sort of i'd recommend build.py options toolbox | 05:24 |
| sdake | rathe rlibvirt | 05:24 |
| sdake | to just build the libvirt container | 05:24 |
| gmmaha | sdake: aah | 05:24 |
| sdake | the rebuild isn't quite right | 05:24 |
| gmmaha | will remember for the next time | 05:25 |
| gmmaha | sdake: is this normal? http://paste.openstack.org/show/492195/ | 05:30 |
| gmmaha | seeing this when the libvirt ontainer is being built | 05:30 |
| sdake | looks wrong | 05:30 |
| sdake | marathon is used by kolla-ansible | 05:30 |
| sdake | rahter kolla-mesos | 05:31 |
| sdake | so I wouldn't be too concerned for your use case | 05:31 |
| gmmaha | also this | 05:32 |
| gmmaha | INFO:__main__:nova-libvirt:invoke-rc.d: policy-rc.d denied execution of start. | 05:32 |
| gmmaha | INFO:__main__:nova-libvirt: | 05:32 |
| sdake | ya nova-libvirt container stryaing to start up in teh buil context | 05:36 |
| sdake | again wouldnt worry about that | 05:36 |
| gmmaha | sdake: aaah | 05:36 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove setuptools and all python.noarch pkgs https://review.openstack.org/298479 | 05:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove setuptools and all python.noarch pkgs https://review.openstack.org/298479 | 05:46 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove setuptools and all python.noarch pkgs https://review.openstack.org/298479 | 05:47 |
| *** sdake_ has joined #kolla | 05:50 | |
| *** sdake has quit IRC | 05:52 | |
| *** tfukushima has joined #kolla | 05:53 | |
| *** weiyu has joined #kolla | 05:57 | |
| sdake_ | mandre nihilifer asalkeld around? | 06:04 |
| sdake_ | pbourke ? | 06:04 |
| sdake_ | gmmaha hows it coming | 06:05 |
| sdake_ | gmmaha i've pretty much hit the wall | 06:05 |
| gmmaha | sdake_: trying to get the file uploaded | 06:05 |
| sdake_ | up since 6am and its midnight | 06:05 |
| gmmaha | sdake_: we can do it tomorrow | 06:05 |
| gmmaha | ouch! | 06:05 |
| gmmaha | thats long! | 06:05 |
| sdake_ | i'm ope nor a couple more hours | 06:05 |
| sdake_ | open | 06:05 |
| gmmaha | we can definitely do it tomorrow | 06:05 |
| sdake_ | tring to get which file uploaded -the strace? | 06:06 |
| *** salv-orlando has joined #kolla | 06:07 | |
| gmmaha | sdake_: trying to upload the docker logs nova_libvirt | 06:08 |
| gmmaha | with strace enabled | 06:08 |
| gmmaha | but unable to get it up to paste.openstack.org via command line | 06:08 |
| sdake_ | email to me if you like | 06:09 |
| sdake_ | stdake@cisco.com | 06:09 |
| gmmaha | sending it your way now | 06:09 |
| *** alisonh has joined #kolla | 06:12 | |
| sdake_ | find the directory in docker for this | 06:16 |
| sdake_ | /usr/lib/x86_64-linux-gnu | 06:16 |
| sdake_ | whre the libvirt-admin so fiel is located | 06:16 |
| *** Marga__ has quit IRC | 06:16 | |
| *** salv-orlando has quit IRC | 06:16 | |
| sdake_ | gmmaha | 06:17 |
| gmmaha | sdake_: checking | 06:17 |
| gmmaha | sdake_: the file is there.. what would you like me to cehck? | 06:18 |
| sdake_ | permissions | 06:18 |
| sdake_ | ls -ld | 06:18 |
| gmmaha | http://paste.openstack.org/show/492196/ | 06:19 |
| *** tuvenen has quit IRC | 06:20 | |
| sdake_ | go up a directory | 06:21 |
| sdake_ | ls -ld | 06:21 |
| gmmaha | root@39bd16befa47:/usr/lib# ls -ld | 06:21 |
| gmmaha | drwxr-xr-x 44 root root 4096 Mar 29 05:29 . | 06:21 |
| sdake_ | up a directory | 06:22 |
| gmmaha | root@39bd16befa47:/usr# ls -ld | 06:22 |
| gmmaha | drwxr-xr-x 25 root root 4096 Mar 29 05:29 . | 06:22 |
| sdake_ | re you in the container itself? | 06:22 |
| gmmaha | sdake_: yeah | 06:22 |
| *** tuvenen has joined #kolla | 06:22 | |
| sdake_ | cd /etc | 06:22 |
| sdake_ | ls -ld ls.* | 06:22 |
| gmmaha | well did the docker run -it .... on libvirt | 06:22 |
| sdake_ | no don't run -it | 06:22 |
| sdake_ | docker exec or check in /var/lib/libvirt | 06:23 |
| sdake_ | rather /var/lib/docker | 06:23 |
| sdake_ | run -it doesn't run with the same docker optiosn as our ansible tools do | 06:23 |
| gmmaha | i cant do docker exec casue the libvirt one is on reboot | 06:23 |
| gmmaha | aaah | 06:23 |
| sdake_ | so use /var/lib/docker | 06:23 |
| sdake_ | use find dto find the right directory | 06:24 |
| sdake_ | and search from there | 06:24 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make kibana container build with debian https://review.openstack.org/297206 | 06:25 |
| gmmaha | sdake_: http://paste.openstack.org/show/492197/ | 06:26 |
| sdake_ | we are getting close to an analysis | 06:26 |
| sdake_ | gmmaha i need to knwo the directory permissions | 06:27 |
| sdake_ | of the parent directories | 06:27 |
| sdake_ | all the way to the top | 06:27 |
| sdake_ | one by one | 06:27 |
| gmmaha | sdake_: yup.. getting it | 06:28 |
| gmmaha | sdake_: http://paste.openstack.org/show/492198/ | 06:29 |
| sdake_ | between the fifth and six,body avity | 06:29 |
| sdake_ | moment | 06:30 |
| sdake_ | ls -ld etc | 06:31 |
| gmmaha | sdake_: didnt follow that.. | 06:31 |
| sdake_ | for the container | 06:31 |
| gmmaha | http://paste.openstack.org/show/492199/ | 06:32 |
| sdake_ | get me the anestaphine from the medlock ! | 06:32 |
| sdake_ | etc/ld.so.conf | 06:33 |
| gmmaha | http://paste.openstack.org/show/492200/ | 06:33 |
| sdake_ | march 17th | 06:34 |
| sdake_ | vs march 29th | 06:34 |
| sdake_ | on your host, run ls -l /etc/ld.so.conf | 06:35 |
| gmmaha | root@jfz1r02h06:/var/lib/docker/aufs/mnt# ls -l /etc/ld.so.conf | 06:35 |
| gmmaha | -rw-r--r-- 1 root root 34 Feb 8 20:41 /etc/ld.so.conf | 06:35 |
| gmmaha | sdake_: just saw that.. hmm... | 06:35 |
| sdake_ | ok we are going to run the container as sit is run by ansible | 06:36 |
| sdake_ | this will tell us which line is bust in the ansible code | 06:36 |
| sdake_ | or atleast give us a pointer | 06:36 |
| gmmaha | neat! | 06:36 |
| *** unicell has joined #kolla | 06:36 | |
| sdake_ | docker run -it --pid=host containername bash | 06:37 |
| sdake_ | then run libvirtd | 06:37 |
| gmmaha | with strace? | 06:38 |
| sdake_ | no | 06:38 |
| sdake_ | justru nit | 06:38 |
| sdake_ | if itworks report "it works" | 06:38 |
| sdake_ | ifit doesnt report "it doesnt" | 06:38 |
| gmmaha | it doesnt | 06:38 |
| gmmaha | without --listen it works | 06:38 |
| gmmaha | with --listen its that stupid CA cert issue | 06:38 |
| sdake_ | fix ca cert | 06:39 |
| sdake_ | on the host, did ou disbble libviert? | 06:40 |
| sdake_ | libvirt? | 06:40 |
| gmmaha | http://paste.openstack.org/show/492201/ | 06:40 |
| gmmaha | it runs, hasnt crashed yet | 06:40 |
| gmmaha | sdake_: yeah | 06:40 |
| * gmmaha goes to confirm | 06:40 | |
| *** xgao has joined #kolla | 06:41 | |
| sdake_ | docker run --it -v /sys/fs/cgroup:/sys/fs/cgroup --pid=host containerid bash | 06:41 |
| gmmaha | shit.. | 06:41 |
| gmmaha | livirtd is running on the host.. :D | 06:42 |
| sdake_ | ya | 06:42 |
| sdake_ | turn that off | 06:42 |
| gmmaha | aaarghh.. i disabled it by think reboot brought it back | 06:42 |
| sdake_ | try again deploy | 06:42 |
| sdake_ | chkconfig disable libvirtd i think will do it | 06:42 |
| sdake_ | although idont know ubuntu so i am not certain | 06:42 |
| gmmaha | sdake_: yeah thats not there in ubuntu but killed it now | 06:43 |
| sdake_ | is your deplo yaio or multinode? | 06:43 |
| gmmaha | multinode | 06:44 |
| sdake_ | make sure libvirt is off on the host on all nodes | 06:44 |
| *** mkoderer has quit IRC | 06:46 | |
| gmmaha | sdake_: disabled on all the hosts and deploying again! | 06:46 |
| sdake_ | make sure to kill running containers | 06:46 |
| * gmmaha goes to add steps to disable libvirt on ubuntu | 06:46 | |
| gmmaha | sdake_: yup.. dumped them | 06:46 |
| sdake_ | les go bck to that strace thing again | 06:46 |
| sdake_ | i need to see what it says now | 06:46 |
| sdake_ | when c code loads, it mmap's the shared object | 06:47 |
| sdake_ | mmap is a system call | 06:47 |
| sdake_ | it can either share or not share | 06:47 |
| sdake_ | in this case its not sharing | 06:47 |
| sdake_ | so that would explainthe permission denied | 06:47 |
| sdake_ | if your /etc/ld.so.conf is being shared from the host | 06:47 |
| sdake_ | i'm not quite sur ewhat ubuntu does with the /etc directory | 06:48 |
| gmmaha | sdake_: aah | 06:48 |
| sdake_ | i am failry convinced its an environmental problem on your end, not a bug | 06:48 |
| gmmaha | i think i am running it with strace | 06:48 |
| sdake_ | but i'll help you sort it out | 06:48 |
| gmmaha | sdake_: if thats the case, and i am sure it worked last week, proabably best to flush the baremetals.. | 06:48 |
| gmmaha | against burning more of your time | 06:48 |
| sdake_ | dont flush baremetal | 06:49 |
| gmmaha | and we debug this on a more condusive scenario | 06:49 |
| *** mkoderer has joined #kolla | 06:49 | |
| sdake_ | there is no need | 06:49 |
| gmmaha | :) | 06:49 |
| sdake_ | at max a reboot would do the job | 06:49 |
| gmmaha | unfortunately that didnt do it | 06:49 |
| gmmaha | that == reboot.. | 06:49 |
| sdake_ | yes need strace | 06:49 |
| gmmaha | the containers are being deployed | 06:50 |
| gmmaha | will get the strace up soon | 06:50 |
| sdake_ | just gie me one strace output | 06:50 |
| sdake_ | i dont need to see the container restarts of all the straces :) | 06:50 |
| gmmaha | :) | 06:52 |
| gmmaha | http://paste.openstack.org/show/492202/ | 06:53 |
| sdake_ | are you running apparmor | 06:54 |
| gmmaha | on the host.. no | 06:55 |
| sdake_ | deployment targets? | 06:55 |
| gmmaha | sdake_: no.. none of them are runnign apparmor | 06:55 |
| sdake_ | here are the things i've targeted on | 06:55 |
| sdake_ | open("/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) | 06:56 |
| sdake_ | open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) | 06:56 |
| sdake_ | change the libvirt --listen to sleep infinity | 06:56 |
| sdake_ | lets look inside the container | 06:57 |
| sdake_ | and see what it has for permissions for those files | 06:57 |
| sdake_ | note normally kolla is not this much of a pain in the ass | 06:57 |
| sdake_ | stat("/lib/x86_64-linux-gnu", 0x7fffd4ac6700) = -1 EACCES (Permission denied) | 06:57 |
| gmmaha | sdake_: i know.. it was a breeze the whole of last week and we deployed atleast 4 times | 06:58 |
| sdake_ | stat("/lib", 0x7fffd4ac6700) = -1 EACCES (Permission denied) | 06:58 |
| gmmaha | guessing i have to change the templates file and re-deploy the libvirt container? | 06:58 |
| sdake_ | yes after killing and removing ghe libvirt container | 06:58 |
| gmmaha | yeah, almost anything and everything is out of bounds on this | 06:58 |
| *** unicell1 has joined #kolla | 06:58 | |
| *** unicell has quit IRC | 07:00 | |
| gmmaha | think i have to throw in the towel now.. i am starting to make super stupid msitakes :| | 07:08 |
| gmmaha | the json was badly formatted.. :( | 07:08 |
| gmmaha | deploy happening now | 07:09 |
| gmmaha | i am in | 07:09 |
| sdake_ | ls -l /etc | 07:11 |
| sdake_ | ls -ld /etc | 07:11 |
| gmmaha | http://paste.openstack.org/show/492203/ | 07:11 |
| sdake_ | is this in the container with sleep infinity that you docker exec'ed into? | 07:12 |
| gmmaha | yup | 07:12 |
| gmmaha | hold on | 07:12 |
| gmmaha | crap | 07:13 |
| gmmaha | sorry yes it was | 07:13 |
| gmmaha | root@jfz1r02h06:~# docker exec -it nova_libvirt bash | 07:13 |
| gmmaha | root@jfz1r02h06:/# | 07:13 |
| gmmaha | wierdly its not showing the docker sha value but just the host name | 07:14 |
| sdake_ | that is inteitonal i think | 07:14 |
| sdake_ | try catting the ld.so.conf file | 07:14 |
| sdake_ | and the cache file | 07:14 |
| *** shardy has joined #kolla | 07:14 | |
| gmmaha | http://paste.openstack.org/show/492204/ | 07:16 |
| sdake_ | gmmaha i'e hit the wall | 07:17 |
| sdake_ | i have some ideas to try in the morning | 07:17 |
| sdake_ | but i need som esleep | 07:17 |
| gmmaha | sdake_: absolutely | 07:17 |
| sdake_ | my ides involve reverting changes to master | 07:17 |
| gmmaha | really apreciate you helping me out on this | 07:17 |
| gmmaha | ouch!! | 07:17 |
| gmmaha | yeah thats defeinintely a tomorrow thing | 07:17 |
| sdake_ | its for th ecommunities benefit as well | 07:17 |
| gmmaha | thanks again for the help | 07:17 |
| gmmaha | absolutely | 07:18 |
| *** sdake_ has quit IRC | 07:22 | |
| *** Serlex has joined #kolla | 07:24 | |
| *** sdake has joined #kolla | 07:25 | |
| *** salv-orlando has joined #kolla | 07:25 | |
| *** yuanying has quit IRC | 07:38 | |
| *** achanda has quit IRC | 07:39 | |
| *** yuanying has joined #kolla | 07:41 | |
| *** achanda has joined #kolla | 07:44 | |
| *** gfidente has joined #kolla | 07:50 | |
| *** achanda has quit IRC | 08:06 | |
| *** mgoddard has joined #kolla | 08:07 | |
| *** achanda has joined #kolla | 08:07 | |
| *** Serlex has quit IRC | 08:08 | |
| *** stvnoyes has quit IRC | 08:11 | |
| *** stvnoyes has joined #kolla | 08:12 | |
| *** achanda has quit IRC | 08:16 | |
| *** pbourke-home has joined #kolla | 08:19 | |
| *** pbourke-home has quit IRC | 08:20 | |
| *** dims has quit IRC | 08:26 | |
| *** salv-orlando has quit IRC | 08:31 | |
| *** dims has joined #kolla | 08:31 | |
| *** jmccarthy has quit IRC | 08:38 | |
| *** jmccarthy has joined #kolla | 08:38 | |
| *** Serlex has joined #kolla | 08:54 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration https://review.openstack.org/291251 | 09:07 |
| *** tfukushima has quit IRC | 09:10 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/298479 | 09:11 |
| *** ccesario__ has joined #kolla | 09:12 | |
| *** tfukushima has joined #kolla | 09:16 | |
| *** achanda has joined #kolla | 09:17 | |
| *** salv-orlando has joined #kolla | 09:20 | |
| openstackgerrit | Merged openstack/kolla: Remove meaningless char in regexp of find_disks.py file https://review.openstack.org/297703 | 09:21 |
| *** achanda has quit IRC | 09:21 | |
| openstackgerrit | Merged openstack/kolla: Only copy the necessary config files for Glance https://review.openstack.org/296982 | 09:26 |
| openstackgerrit | Merged openstack/kolla: Refactor the glance service group mapping https://review.openstack.org/298100 | 09:27 |
| *** Liuqing has quit IRC | 09:47 | |
| *** salv-orlando has quit IRC | 09:57 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the Ceph reconfigure https://review.openstack.org/298635 | 10:07 |
| openstackgerrit | Andrey Pavlov proposed openstack/kolla-mesos: Using oslo_log for logging in CLI https://review.openstack.org/298231 | 10:09 |
| *** salv-orlando has joined #kolla | 10:10 | |
| *** sdake has quit IRC | 10:10 | |
| *** pbourke has quit IRC | 10:16 | |
| *** allen_gao has left #kolla | 10:16 | |
| *** pbourke has joined #kolla | 10:17 | |
| *** ccesario__ has quit IRC | 10:23 | |
| *** Liuqing has joined #kolla | 10:27 | |
| openstackgerrit | Merged openstack/kolla: Remove bootstrap_mariadb with cleanup-containers https://review.openstack.org/298328 | 10:31 |
| *** banix has joined #kolla | 10:34 | |
| *** tfukushima has quit IRC | 10:36 | |
| *** kproskurin has joined #kolla | 10:47 | |
| kproskurin | Hi guys, does anyone tried to rebuild horizon and\or keystone centos containers? | 10:48 |
| kproskurin | I have a “Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_64" | 10:48 |
| kproskurin | During a build | 10:48 |
| kproskurin | error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file | 10:49 |
| *** banix has quit IRC | 10:49 | |
| kproskurin | I google it and it somekind of all docker hub base image problem | 10:49 |
| kproskurin | s/all/old/ | 10:49 |
| *** achanda has joined #kolla | 10:50 | |
| *** achanda has quit IRC | 10:55 | |
| *** xgao has quit IRC | 10:55 | |
| *** Liuqing has quit IRC | 10:56 | |
| *** banix has joined #kolla | 11:00 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 11:07 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make kibana container build with debian https://review.openstack.org/297206 | 11:07 |
| kproskurin | https://github.com/docker/hub-feedback/issues/461 | 11:09 |
| *** weiyu has quit IRC | 11:12 | |
| akwasnie | kproskurin: I have the same issue | 11:13 |
| kproskurin | Docker guys reaction time is just great | 11:16 |
| ccesario | morning! | 11:19 |
| *** Serlex has left #kolla | 11:20 | |
| ccesario | kproskurin, look this comment... https://github.com/docker/docker/issues/8966 | 11:20 |
| ccesario | This is not a bug in Docker, but an issue with older AUFS versions missing CONFIG_AUFS_XATTR support. To resolve the issue, update the AUFS version and enable the CONFIG_AUFS_XATTR attribute, or switch to another storage driver. | 11:20 |
| kproskurin | Well, I have kernel 4.2.0-25, no one said in this comments which kernel is needed. Or aufs-tools needed to be updated?.. | 11:23 |
| ccesario | hmmmm | 11:23 |
| kproskurin | But # CONFIG_AUFS_XATTR is not set | 11:24 |
| kproskurin | in kernel | 11:24 |
| ccesario | hmmmm | 11:24 |
| ccesario | let me do some search regarding this... (I ḿ not a CentOS user) | 11:24 |
| kproskurin | My complain is what it worked some time before. so it’s not right to say “it’s not a docker problem” | 11:24 |
| *** banix has quit IRC | 11:25 | |
| kproskurin | It is, since it worked on the same kernel before :-D | 11:25 |
| kproskurin | My host system is Ubuntu 14.04 actually with Wily kernel | 11:25 |
| ccesario | Ubuntu?! O_o | 11:26 |
| kproskurin | Ubuntu! | 11:26 |
| *** Marga_ has joined #kolla | 11:26 | |
| kproskurin | You said it like “Ubuntu? wtf, it s piece of shit…” :-) | 11:27 |
| ccesario | kproskurin, it is the same me... | 11:27 |
| ccesario | but I dont see errors in my deploy/setup | 11:28 |
| ccesario | docker --version | 11:29 |
| ccesario | Docker version 1.10.0, build 590d5108 | 11:29 |
| kproskurin | Docker version 1.10.2, build c3959b1 | 11:31 |
| kproskurin | ccesario: what is your host OS? | 11:31 |
| kproskurin | imo it’s ubuntu kernel related | 11:32 |
| ccesario | Description:Ubuntu 14.04.4 LTS - Linux operator 4.2.0-30-generic #36~14.04.1-Ubuntu SMP Fri Feb 26 18:49:23 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | 11:33 |
| *** jtriley has joined #kolla | 11:34 | |
| kproskurin | ccesario: sooo, you can build cenos base keystone & horizon on this host? | 11:34 |
| *** Liuqing has joined #kolla | 11:35 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a step to clean all ceph references in fstab https://review.openstack.org/298363 | 11:35 |
| ccesario | kproskurin, ohhh centos ? ... no no... my containers are ubuntu based | 11:35 |
| kproskurin | :-D | 11:35 |
| kproskurin | It’s centos base related problem | 11:35 |
| *** rhallisey has joined #kolla | 11:36 | |
| kproskurin | kolla-build —base centos horizon | 11:36 |
| kproskurin | -s devicemapper as a workaround works just fine :-\ | 11:40 |
| ccesario | kproskurin, http://paste.openstack.org/show/492244/ | 11:41 |
| ccesario | It is only a short test | 11:42 |
| ccesario | according this comments the problem happen when centos is deployed over ubuntu host https://github.com/docker/docker/issues/6980 | 11:43 |
| *** Marga_ has quit IRC | 11:54 | |
| *** dwalsh has joined #kolla | 11:59 | |
| *** Serlex has joined #kolla | 12:05 | |
| *** salv-orlando has quit IRC | 12:09 | |
| *** salv-orlando has joined #kolla | 12:17 | |
| *** SiRiuS_ has joined #kolla | 12:17 | |
| *** vhosakot has joined #kolla | 12:18 | |
| *** pbourke-home has joined #kolla | 12:22 | |
| openstackgerrit | Merged openstack/kolla-mesos: Bump rabbitmq version in rabbitmq-env.conf https://review.openstack.org/298165 | 12:23 |
| openstackgerrit | Merged openstack/kolla-mesos: Override Mesos config in configuration tests https://review.openstack.org/296924 | 12:23 |
| *** pbourke-home has quit IRC | 12:24 | |
| openstackgerrit | Merged openstack/kolla-mesos: Encode the ifname before passing to struct https://review.openstack.org/295829 | 12:27 |
| openstackgerrit | Merged openstack/kolla-mesos: Fix string parameter in mesos-dns.conf https://review.openstack.org/295830 | 12:27 |
| *** jtriley has quit IRC | 12:31 | |
| *** jtriley has joined #kolla | 12:34 | |
| *** jtriley has quit IRC | 12:39 | |
| openstackgerrit | Merged openstack/kolla-mesos: Parse and render yaml files in the proper order https://review.openstack.org/295831 | 12:42 |
| *** rhallisey has quit IRC | 12:44 | |
| *** rhallisey has joined #kolla | 12:44 | |
| *** weiyu_ has joined #kolla | 12:46 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Fix rabbitmq version for centos https://review.openstack.org/298720 | 12:49 |
| *** achanda has joined #kolla | 12:52 | |
| *** weiyu_ has quit IRC | 12:53 | |
| *** dwalsh has quit IRC | 12:55 | |
| *** achanda has quit IRC | 12:58 | |
| *** klint has quit IRC | 12:59 | |
| *** jtriley has joined #kolla | 13:01 | |
| *** banix has joined #kolla | 13:09 | |
| *** ayoung has joined #kolla | 13:09 | |
| *** jtriley has quit IRC | 13:12 | |
| *** dwalsh has joined #kolla | 13:14 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Move memcached deps to bootstrap section for horizon https://review.openstack.org/298741 | 13:24 |
| *** vhosakot has quit IRC | 13:25 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Move memcached deps to bootstrap section for horizon https://review.openstack.org/298741 | 13:27 |
| SiRiuS_ | kproskurin, do you know how I should set up kolla-messos.conf for a multi-node setup? | 13:32 |
| SiRiuS_ | kproskurin, i have 3 masters, 3 controllers and 1 compute, just like the Vagrantfile crates | 13:33 |
| *** absubram has joined #kolla | 13:34 | |
| SiRiuS_ | kproskurin, the kolla-mesos has only one host per service section | 13:34 |
| *** absubram_ has joined #kolla | 13:35 | |
| kproskurin | SiRiuS_: Yes, there is some not intuitive things and some bug in upstream right now. But right now im on meeteeng, could you pls wait 20-30 min? | 13:36 |
| SiRiuS_ | kproskurin, sure thing :) | 13:38 |
| *** jtriley has joined #kolla | 13:39 | |
| *** absubram has quit IRC | 13:40 | |
| *** absubram_ is now known as absubram | 13:40 | |
| *** SiRiuS_ has quit IRC | 13:43 | |
| *** diogogmt has quit IRC | 13:43 | |
| *** mgoddard_ has joined #kolla | 13:44 | |
| *** SiRiuS_ has joined #kolla | 13:46 | |
| *** diogogmt has joined #kolla | 13:47 | |
| *** mgoddard has quit IRC | 13:47 | |
| *** inc0 has joined #kolla | 13:55 | |
| inc0 | good morning | 13:55 |
| *** mkoderer__ has quit IRC | 13:59 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make kibana container build with debian https://review.openstack.org/297206 | 14:00 |
| *** Liuqing has quit IRC | 14:02 | |
| *** mkoderer__ has joined #kolla | 14:03 | |
| kproskurin | SiRiuS_: Alright, so, sorry for lack of documentation right now, kolla-mesos changes too rapidly. And once side node - I dont use vagrant stuff, so we need to re-check is you mesos stack is working actually | 14:04 |
| *** absubram has quit IRC | 14:04 | |
| kproskurin | SiRiuS_: I understand right, what you have a marathon,mesos,zookeeper already running? | 14:05 |
| *** diogogmt has quit IRC | 14:05 | |
| SiRiuS_ | kproskurin, yes | 14:05 |
| SiRiuS_ | kproskurin, I already deployed those | 14:05 |
| kproskurin | Alright, so you have 3 controller nodes and 1 compute, right? | 14:06 |
| SiRiuS_ | kproskurin, I have 5 services running on the masters | 14:06 |
| SiRiuS_ | kproskurin, And I have mess slave running on the controllers and compute nods | 14:06 |
| SiRiuS_ | kproskurin, yes, 3 controllers and 1 compute | 14:06 |
| kproskurin | SiRiuS_, Alright, for multinode you need to check for mesos attributes. Your mesos stack runs in containers? | 14:08 |
| *** dwalsh has quit IRC | 14:08 | |
| SiRiuS_ | kproskurin, http://paste.openstack.org/show/492280/ | 14:09 |
| SiRiuS_ | that is on one of the master nodes | 14:09 |
| kproskurin | Ok, I just looked into code, and looks like we set attributes for mesos-slave, which is good. I was afraid we forgot to add it | 14:10 |
| kproskurin | So, right now, you need to do some stuff | 14:10 |
| kproskurin | 1) Pull latest master from git | 14:10 |
| kproskurin | 2) Cherry-pick this patches:https://review.openstack.org/#/c/298741/ https://review.openstack.org/#/c/298720/ https://review.openstack.org/#/c/291251/ | 14:11 |
| patchbot | kproskurin: patch 298741 - kolla-mesos - Move memcached deps to bootstrap section for horizon | 14:11 |
| patchbot | kproskurin: patch 298720 - kolla-mesos - Fix rabbitmq version for centos | 14:11 |
| patchbot | kproskurin: patch 291251 - kolla-mesos - Fix OpenStack services registration | 14:11 |
| kproskurin | oh, I forgot we have heka stuff broken right now. :-\ | 14:12 |
| inc0 | kproskurin, heka stuff broken? you mean mesos? | 14:13 |
| inc0 | or ansible is broken somewhow? | 14:13 |
| kproskurin | SiRiuS_ So as a temp workaround(we fixing it right now), pls apply this http://fpaste.org/346991/45926076/ AND rebuild kolla-toolbox with this http://fpaste.org/346992/92608181/ - this is a mess, sorry for that, we’re working on a fix | 14:14 |
| kproskurin | inc0: we dont yet implement all this heka stuff, but already add patch with heka logs dirs and such | 14:14 |
| kproskurin | So we have some permission problems right now :-\ | 14:14 |
| SiRiuS_ | kproskurin, And after I apply all the fixes, how to I edit the kolla-mesos.conf file ? | 14:16 |
| SiRiuS_ | kproskurin, I has just one host attribute per service | 14:16 |
| SiRiuS_ | kproskurin, but I have 3 hosts for master instead of 1 | 14:16 |
| kproskurin | SiRiuS_: it’s ok. Ill paste my configs, 1 min | 14:17 |
| SiRiuS_ | ok | 14:17 |
| kproskurin | SiRiuS_: for kolla-mesos.conf you basicly need to edit you mesos stack ips: http://fpaste.org/346994/92610621/ | 14:18 |
| SiRiuS_ | kproskurin, I see I can set multiple IPs for the host attribute | 14:20 |
| SiRiuS_ | kproskurin, separated by commas | 14:21 |
| kproskurin | SiRiuS_: its more triky with /etc/kolla-mesos/globals.yml, my example: http://fpaste.org/347000/26131014/ . So, you have to check all docker vas, like registry and set your network eth and set right mesos_dns domain(from mesos-dns config in container), etc, etc | 14:22 |
| kproskurin | SiRiuS_: cont think about constrains for now, it should detect your mesos-slaves nodes and fetch their attributes and use it for constrains | 14:23 |
| *** dwalsh has joined #kolla | 14:24 | |
| kproskurin | After all this, you should be able to deploy multinode…. but we have yet one more bug with mariadb bootstrap in multinode, Ill tell you how to workaround it if you get there :-) | 14:25 |
| *** pbourke-home has joined #kolla | 14:25 | |
| kproskurin | We actually working on fixing all bugs right now and hope ti finish at this week | 14:25 |
| kproskurin | After that we probably will update docs | 14:26 |
| kproskurin | Right now its a mess, sorry | 14:26 |
| SiRiuS_ | kproskurin, cool, I'll also be adding/fixing a couple of Vagrant bugs | 14:26 |
| *** pbourke-home has quit IRC | 14:26 | |
| SiRiuS_ | kproskurin, like getting the hostmanager to work correctly with VirtualBox | 14:27 |
| SiRiuS_ | kproskurin, and using the latest docker registry, registry:2 | 14:27 |
| *** sdake has joined #kolla | 14:27 | |
| kproskurin | SiRiuS_: Great! Post a patches if you catch something nasty, sadly we dont really use vagrant right now, so it could be in a bad state | 14:27 |
| inc0 | sdake, so I'm testing deploying kolla mitaka with liberty code | 14:28 |
| SiRiuS_ | kproskurin, Vagrant is good for reproducible environments | 14:28 |
| inc0 | liberty services | 14:28 |
| inc0 | and so far keystone needs love | 14:28 |
| inc0 | due to fernet tokens | 14:28 |
| inc0 | or no...well due to something | 14:28 |
| inc0 | but KS needs love | 14:28 |
| *** sdake_ has joined #kolla | 14:30 | |
| *** sdake has quit IRC | 14:32 | |
| *** achanda has joined #kolla | 14:33 | |
| sdake_ | morning | 14:37 |
| *** sdake_ is now known as sdake | 14:37 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Move memcached deps to bootstrap section for horizon https://review.openstack.org/298741 | 14:39 |
| *** dwalsh has quit IRC | 14:40 | |
| inc0 | hey | 14:40 |
| inc0 | sdake, | 14:40 |
| sdake | inc0 sup cat | 14:41 |
| inc0 | ks won't work with mitaka deploy and liberty code | 14:41 |
| inc0 | but change should be fairly small | 14:41 |
| sdake | what is ks | 14:41 |
| inc0 | keystone | 14:41 |
| sdake | forigve,j ust woek up | 14:41 |
| sdake | caffeine free diet ftl | 14:41 |
| *** kproskurin has quit IRC | 14:49 | |
| *** kproskurin has joined #kolla | 14:49 | |
| *** mgoddard_ has quit IRC | 14:52 | |
| *** mgoddard has joined #kolla | 14:53 | |
| *** dwalsh has joined #kolla | 14:53 | |
| *** haplo37 has joined #kolla | 14:54 | |
| sdake | groan my gate job failed | 15:01 |
| sdake | and i specifically tested centos source | 15:01 |
| openstackgerrit | Andrei-Lucian Șerb proposed openstack/kolla-mesos: Set correct NIC number for hostmanager IP resolver https://review.openstack.org/298805 | 15:04 |
| *** openstackgerrit has quit IRC | 15:06 | |
| *** openstackgerrit has joined #kolla | 15:06 | |
| *** diogogmt has joined #kolla | 15:11 | |
| sdake | pbourke around? | 15:15 |
| *** diogogmt has quit IRC | 15:17 | |
| *** diogogmt has joined #kolla | 15:18 | |
| *** vhosakot has joined #kolla | 15:18 | |
| pbourke | sdake: hey | 15:21 |
| *** salv-orl_ has joined #kolla | 15:21 | |
| *** haplo37 has quit IRC | 15:23 | |
| *** salv-orlando has quit IRC | 15:24 | |
| inc0 | pbourke, so...I need to have discussion with you | 15:27 |
| inc0 | brace for impact | 15:27 |
| inc0 | it's about backporting | 15:27 |
| inc0 | I took over this issue | 15:27 |
| inc0 | and we are looking at horrible scope creep | 15:27 |
| inc0 | I tried to run and it turns out that we can | 15:27 |
| inc0 | can't easily deploy rsyslog with kolla docker and docker 1.10 | 15:28 |
| inc0 | so we either hack around it or backport heka | 15:28 |
| inc0 | and since heka is way better, I'd go for backport heka | 15:28 |
| inc0 | and then we are looking at backporting kolla_docker, thin containers, upgrades and now heka | 15:29 |
| inc0 | also we were taling about reconfigure | 15:29 |
| *** dwalsh has quit IRC | 15:29 | |
| inc0 | so...that's mitaka worth of work | 15:29 |
| inc0 | I want to make mitaka kolla deploy liberty openstack at this point. Will be way more stable than what we were going to do | 15:29 |
| inc0 | I know you were strongly opposed to this. Can we discuss? | 15:30 |
| pbourke | sure | 15:30 |
| inc0 | I honestly thing amount of hacking to make backports work will be impossible to make in any stable shape and form | 15:31 |
| inc0 | think | 15:32 |
| inc0 | tell me this, are you still opposed to this? | 15:32 |
| pbourke | not really | 15:32 |
| gmmaha | sdake: good morning :) | 15:34 |
| vhosakot | instead of touching the stable/mitaka branch, does it make sense to do all the backport-to-mitaka work in a separate branch like mitaka-backport, and then merge stable/mitaka with mitaka-backport once mitaka-backport is stable and tested ? do other OpenStack projects do such a thing ? | 15:35 |
| sdake | gmmaha hey bro | 15:35 |
| sdake | vhosakot the answer is not really | 15:35 |
| sdake | vhosakot branches are permanent things | 15:35 |
| sdake | there are feature branches | 15:35 |
| sdake | and I can have them deleted | 15:35 |
| sdake | but its atypical to use them in this way in openstack | 15:36 |
| inc0 | and the issue is...well it's 90% of config options and 10% of twaks | 15:37 |
| inc0 | tweaks | 15:37 |
| inc0 | and I hope I can get exactly same code working for both cases and making it 100% config options | 15:37 |
| gmmaha | sdake: think you would have some time this morning for the patches you wanted to test with a revert | 15:37 |
| vhosakot | ah ok | 15:37 |
| pbourke | inc0: why do we need docker 1.10 for liberty | 15:37 |
| *** blahRus has joined #kolla | 15:38 | |
| sdake | gmmaha i 've got some emails to write | 15:38 |
| sdake | gmmaha ca you give me 30 misn to 1 hour | 15:38 |
| sdake | just getting my day going | 15:38 |
| gmmaha | sdake: ohh for sure.. take all the time | 15:39 |
| gmmaha | no rush.. | 15:39 |
| gmmaha | thanks | 15:39 |
| inc0 | pbourke, thin containers | 15:39 |
| inc0 | we need 1.9 for volumes | 15:39 |
| inc0 | and 1.10 for thin containers | 15:39 |
| sdake | manjeets ping | 15:40 |
| pbourke | we dont *have* to have thin containers in liberty though | 15:40 |
| sdake | gmmaha are you sufering from the mariadb lights out recovery bug? | 15:41 |
| vhosakot | gmmaha: i'm around if you'd like to dig mariadb.. I'm trying multinode | 15:41 |
| gmmaha | sdake: dont think i have hit that bug | 15:41 |
| inc0 | pbourke, but we were testing kolla_docker with 1.10 almost all the time | 15:41 |
| sdake | vhosakot we were working on libvirt | 15:41 |
| inc0 | it should work with 1.9 | 15:41 |
| gmmaha | vhosakot: sdake: hit that issue when i tried an upgrade against a deploy | 15:41 |
| gmmaha | thats the bug i had filed | 15:41 |
| sdake | gmmaha link to bug plz | 15:41 |
| * gmmaha goes to find it | 15:42 | |
| inc0 | and getting features cut out like that will be hard | 15:42 |
| vhosakot | ah libvirt permission issue for libvirt.so | 15:42 |
| sdake | ok - email | 15:42 |
| sdake | then i'll be back | 15:42 |
| vhosakot | cool | 15:42 |
| sdake | actually i have a team meeting | 15:42 |
| sdake | then I'll be back | 15:42 |
| gmmaha | sdake: vhosakot: https://bugs.launchpad.net/kolla/+bug/1561132 | 15:42 |
| openstack | Launchpad bug 1561132 in kolla mitaka "Kolla upgrade on multi-node with master fails" [Critical,Triaged] - Assigned to Michał Jastrzębski (inc007) | 15:42 |
| inc0 | so what we could do is to find a commit right after kolla_docker was merged | 15:42 |
| sdake | my team meeting is at 10pst | 15:42 |
| inc0 | and try to backport from there | 15:42 |
| inc0 | but it was always 1.10 tested | 15:42 |
| inc0 | yay | 15:43 |
| sdake | inc0 need to ge tbuggs off yoru plate :( | 15:44 |
| vhosakot | gmmaha: did you get a chance to start libvirtd in debug mode after adding -v flag ? | 15:44 |
| sdake | vhosakot we ran it through strace | 15:44 |
| *** haplo37 has joined #kolla | 15:44 | |
| inc0 | yeah, and I didn't write mariadb upgrade | 15:44 |
| sdake | strace has a bunch of permission denied | 15:44 |
| pbourke | inc0: I say go for it, it sounds like you've looked at this in more detail than I have. If you can make it happen I don't have a problem | 15:44 |
| sdake | vhosakot go read the logs from last night to catch up pleae | 15:44 |
| sdake | could use your help debugging this one | 15:44 |
| vhosakot | yep... | 15:44 |
| *** dwalsh has joined #kolla | 15:44 | |
| inc0 | pbourke, yeah it seems like it, I wanted to make sure that you're o board with it | 15:44 |
| sdake | we will have a formal vote | 15:45 |
| inc0 | will talk about it tomorrow on meeting | 15:45 |
| sdake | but with pbourke's spport I expect the vote to pass | 15:45 |
| inc0 | sdake, I won't wait to vote, there is good chance it will pass and we have super little time | 15:45 |
| sdake | since pbourke/sdake were most vocl criticals | 15:45 |
| sdake | inc0 dont wait on vote | 15:45 |
| inc0 | worst case scenerio, I waste time | 15:45 |
| gmmaha | vhosakot: yeah we did that and even started the app with an strace. it seems like the permissions are amiss and that prevents the files from being read | 15:45 |
| sdake | if people vote no and it doesnt have majority | 15:45 |
| sdake | i'll ask the people that voted no to fix it instead ;) | 15:45 |
| inc0 | lol | 15:46 |
| inc0 | thats why votings are anonymus | 15:46 |
| sdake | vhosakot readl ogs (in irc tpic) | 15:46 |
| sdake | from yeterday to see the debuggin we went through | 15:46 |
| sdake | i am totally stumped | 15:46 |
| sdake | but i just woke up | 15:46 |
| sdake | and have meails tow rite! | 15:46 |
| sdake | bbiaf | 15:46 |
| vhosakot | gmmaha: sdake: yep, reading the IRC logs.. few mins | 15:47 |
| gmmaha | vhosakot: thanks | 15:47 |
| sdake | gmmaha my day is packed with meetings | 15:48 |
| sdake | gmmaha i'm not uure how much bandwidth i can provide you today until after 3;30 PST | 15:48 |
| sdake | after hiwch i am free | 15:48 |
| gmmaha | sdake: aah no worries.. | 15:48 |
| sdake | manjeets when you get in ing me | 15:48 |
| gmmaha | if i dont get anywhere we can do it later today | 15:48 |
| gmmaha | sdake: i will let him know. | 15:48 |
| gmmaha | he is usually in around 9ish | 15:48 |
| sdake | wow inc0 | 15:53 |
| sdake | from the ops summit meeting schedule | 15:53 |
| sdake | Is Kolla a universal solution for this difficulty? | 15:53 |
| inc0 | yeah I saw | 15:54 |
| sdake | difficulty being | 15:54 |
| inc0 | for packaging | 15:54 |
| sdake | Deployment methods - OS distro packages, Ansible + containers/tarballs/whatever, homegrown packages, Kolla, puppet, Ansible, others? Let's discuss the pros and cons of these so we can be educated with our decisions moving forward, particularly: | 15:54 |
| inc0 | we definetely need to be on these sessions | 15:54 |
| sdake | kolla folks please vote https://etherpad.openstack.org/p/AUS-ops-meetup | 15:54 |
| sdake | inc0 yes i modified my travel arrangements just to hit the ops summit | 15:55 |
| sbezverk | have you ever seen rabbitmq container restarting with this error: {error_logger,{{2016,3,29},{16,41,53}},"Too short cookie string",[]} | 15:55 |
| sdake | sbezverk /me groans | 15:55 |
| inc0 | I'll be in Austin since Sunday | 15:55 |
| sdake | comon guys yur killing me :) | 15:55 |
| sdake | inc0 me too | 15:55 |
| vhosakot | gmmaha: still in IRC logs.. will ping when I'm done | 15:56 |
| sbezverk | sdake are we filing new bug? since it is repoducible? | 15:57 |
| inc0 | sbezverk, if it is reproducible, yes | 15:59 |
| gmmaha | vhosakot: take your time.. | 16:00 |
| sbezverk | inc0 it looks like there are already two lucky guys hitting it, sdake and me :-) | 16:00 |
| sbezverk | I asked for some help on oslo IRC | 16:00 |
| *** dwalsh has quit IRC | 16:02 | |
| openstackgerrit | Andrei-Lucian Șerb proposed openstack/kolla-mesos: Add user vagrant to group docker https://review.openstack.org/298848 | 16:06 |
| vhosakot | gmmaha: so, is this the logs after running libvirt with strace enabled... http://paste.openstack.org/show/492202/ | 16:07 |
| gmmaha | vhosakot: yeah | 16:07 |
| vhosakot | gmmaha: should I run with strace do check how a good output looks ? | 16:08 |
| vhosakot | do you think it is a good test ? | 16:08 |
| gmmaha | vhosakot: definitely think so | 16:08 |
| gmmaha | was tyring to see if i can bring up an AIO for the same | 16:08 |
| *** dwalsh has joined #kolla | 16:08 | |
| vhosakot | gmmaha: those "permission denied | 16:09 |
| vhosakot | for stat and open | 16:09 |
| vhosakot | ok | 16:09 |
| vhosakot | gmmaha: https://www.redhat.com/archives/libvirt-users/2014-May/msg00017.html | 16:10 |
| sdake | gmmaha I will be with you from 3:30-5pm | 16:10 |
| sdake | gmmaha after 5pm I am booked using manjeets gear to diagnose a reproducible critical issue | 16:11 |
| gmmaha | sdake: for sure.. | 16:11 |
| gmmaha | if i cant figure out betweeen now and 3:30 | 16:11 |
| gmmaha | sdake: i might be away around 3ish... let me see how much i can get done before then | 16:12 |
| vhosakot | gmmaha: you are on docker 1.10 right ? docker -v | 16:12 |
| sdake | was there a followup to tha tthread? | 16:12 |
| sdake | vhosakot 10.0.3 | 16:13 |
| vhosakot | sdake: no.. I see no followup | 16:13 |
| gmmaha | Docker version 1.10.3, build 20f81dd | 16:13 |
| vhosakot | gmmaha: ok | 16:13 |
| sbezverk | sdake have you managed to workaournd rabbitmq issue? | 16:15 |
| sdake | gmmaha qucik q, which kernel arey ou using on your deploy targets | 16:15 |
| gmmaha | sdake: 3.13.0-77-generic | 16:15 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 16:16 |
| *** Serlex has quit IRC | 16:18 | |
| gmmaha | vhosakot: the commands per the thread work fine within the container | 16:18 |
| gmmaha | when i try and run stat and open | 16:18 |
| vhosakot | yep that is the difference even I see... 1 sec | 16:18 |
| gmmaha | vhosakot: aah | 16:18 |
| sdake | is someone reigstered with nickserv tha tcan ask questions in docker-dev | 16:19 |
| sdake | i can't find my password | 16:19 |
| sdake | and we need to involve the docker dev community in this bug | 16:19 |
| sdake | gmmaha i dont know what version that is | 16:19 |
| sdake | gmmaha did you run apt-get install linux-image-generic-lts-wily | 16:20 |
| gmmaha | sdake: dont think so.. | 16:20 |
| sdake | holy christ | 16:20 |
| gmmaha | ntpttr: did you? | 16:20 |
| vhosakot | sdake: I am in docker-dev everyday... what exactly should we ask... gmmaha: did you want to frame a question ? | 16:20 |
| sdake | well that is what the problem is | 16:20 |
| sdake | vhosakot lets put a hold on that | 16:20 |
| gmmaha | sdake: sorry not sure if we did.. ntpttr did the intial setup and he might know | 16:20 |
| sdake | vhosakot what is your kernel version | 16:21 |
| openstackgerrit | MD NADEEM proposed openstack/kolla: Add optional services option in globals.yml https://review.openstack.org/298104 | 16:21 |
| ntpttr | gmmaha: sdake: yes we ran that command | 16:21 |
| sdake | vhosakot find your kernel version lets see if it matches | 16:21 |
| sdake | ntpttr on all of the taget deployment nodes? | 16:22 |
| ntpttr | sdake: yes | 16:22 |
| vhosakot | 4.2.0-30-generic | 16:22 |
| sdake | well vhosakot's kernel and yours differ | 16:22 |
| sdake | vhosakot what kernel are you running | 16:22 |
| sdake | 3.13 is super old | 16:22 |
| sdake | containers barely run in 3.13 | 16:22 |
| vhosakot | isn't LTS 4.2 a requirement for kolla ? | 16:22 |
| sdake | unless its centos 3.10 where redhat backports everthing but doesn't change the version # | 16:22 |
| gmmaha | sdake: hmm.. | 16:23 |
| sdake | what is lts 4.2 | 16:23 |
| sdake | ok, guys | 16:23 |
| sdake | I dont know anything about ubuntu | 16:23 |
| sdake | so when i ask dumb questions like what is 4.2 | 16:23 |
| inc0 | vhosakot, yeah, you need 4.2 | 16:23 |
| sdake | please bear with me, I'm trying to learn | 16:23 |
| inc0 | sdake, kernel | 16:23 |
| sdake | trying/am learning | 16:23 |
| ntpttr | hmm.. I'm not sure if/how they might have been reverted, but last week when we had a functional deployment I'm sure that I had run that update | 16:23 |
| gmmaha | let me reboot the machine and see if i can force it to boot 4.2 kernel | 16:23 |
| sdake | inc0 so our docs are wrong? | 16:23 |
| inc0 | no | 16:24 |
| vhosakot | gmmaha: can you do this ? | 16:24 |
| vhosakot | $ dpkg -l | grep linux-image-generic-lts-wily | 16:24 |
| sdake | ok so gmmaha has wrong kernel | 16:24 |
| inc0 | our docs states "install wily" | 16:24 |
| inc0 | which is 4.2 | 16:24 |
| vhosakot | yes.. wily wily wily... step 1 | 16:24 |
| vhosakot | gmmaha: can you do $ dpkg -l | grep linux-image-generic-lts-wily ? | 16:24 |
| *** dwalsh has quit IRC | 16:24 | |
| gmmaha | vhosakot: nope.. empty string | 16:25 |
| gmmaha | :( | 16:25 |
| sdake | kernel verison should be in prechecks!! | 16:25 |
| *** Marga_ has joined #kolla | 16:25 | |
| vhosakot | gmmaha: empty!! | 16:25 |
| gmmaha | sdake: we always ran prechecks when i installed | 16:25 |
| gmmaha | vhosakot: dpkg -l | grep linux-image-generic-lts-wily | 16:26 |
| gmmaha | root@jfz1r02h07:~# | 16:26 |
| sdake | can someone file a bug for that, i have meeting in 4 minutes | 16:26 |
| vhosakot | gmmaha: meet me at the bar :) | 16:26 |
| sdake | gmmaha yes but kernel version is not in prechecks | 16:26 |
| vhosakot | sdake: I will file one | 16:26 |
| gmmaha | vhosakot: :) | 16:26 |
| gmmaha | sdake: aah | 16:26 |
| sdake | vhosakot thnkas | 16:26 |
| inc0 | vhosakot, remember to keep in mind it may be different in centos | 16:26 |
| gmmaha | sorry for the red herring | 16:26 |
| gmmaha | :| | 16:26 |
| vhosakot | gmmaha: did the same host ever work for kolla ? | 16:26 |
| gmmaha | vhosakot: all these hosts did | 16:27 |
| gmmaha | which is what is puzzling | 16:27 |
| gmmaha | they all worked until last friday | 16:27 |
| inc0 | it kinda is | 16:27 |
| vhosakot | inc0: right, the precehcks need to cehck wily just on Ubuntu host... does this make sense ? | 16:27 |
| sdake | gmmaha you rebuilt yoru machines | 16:27 |
| sdake | you said you wnt back to bare metal | 16:27 |
| gmmaha | sdake: nope.. | 16:27 |
| inc0 | vhosakot, well, you can do this with ansible, ansible provides this kind of stiff | 16:27 |
| inc0 | and precheck just for buntu is better than no precheck at all imho | 16:27 |
| gmmaha | been holding off while we were trying to debug this | 16:27 |
| ntpttr | it was always bare metal | 16:27 |
| vhosakot | prechecks is already playbook I see | 16:27 |
| sdake | centos needsn o precheck | 16:28 |
| sdake | centos 7 will awlays work | 16:28 |
| ntpttr | :q | 16:28 |
| ntpttr | oops sorry | 16:28 |
| sdake | ok well lets fix that | 16:28 |
| sdake | and go from there | 16:28 |
| vhosakot | gmmaha: http://docs.openstack.org/developer/kolla/quickstart.html#installing-dependencies... | 16:28 |
| sdake | because that could cuase the wierdness you see | 16:28 |
| gmmaha | sdake: yes.. getting the new kernel istnstalled | 16:29 |
| vhosakot | gmmaha: I suggest you run "apt-get install linux-image-generic-lts-wily" reboot the host and redeploy | 16:29 |
| sdake | i recommend additional step | 16:29 |
| sdake | stop docker daemon | 16:29 |
| sdake | wipe out /var/lib/docker | 16:29 |
| sdake | start docker daemon | 16:29 |
| sdake | just so we don have left over stuff | 16:29 |
| gmmaha | sdake: for sure | 16:29 |
| sdake | this is the nuclear otpoin with docker removals ;) | 16:30 |
| sdake | rathe rcleanups | 16:30 |
| vhosakot | does "apt-get install linux-image-generic-lts-wily" take effect without reboot ? if yes, I will add this in precehecks | 16:30 |
| gmmaha | vhosakot: i doubt so.. | 16:31 |
| gmmaha | you will need a reboot | 16:31 |
| vhosakot | gmmaha: right.. since it is a kernel update | 16:31 |
| *** Marga_ has quit IRC | 16:31 | |
| gmmaha | but a warning saying you shall not pass without this in prechecks is mighty awesome! :) | 16:31 |
| vhosakot | yep | 16:32 |
| vhosakot | inc0: so, this needs update to cehck TLS on UBuntu ? https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/package_checks.yml | 16:32 |
| vhosakot | LTS* i meant | 16:32 |
| inc0 | I would say so | 16:33 |
| vhosakot | inc0: cool, thanks | 16:33 |
| inc0 | however, keep in mind that it has to run on every host | 16:33 |
| *** banix_ has joined #kolla | 16:33 | |
| inc0 | so it's like docker check task | 16:33 |
| gmmaha | inc0: doesnt precheck run on every host when you do a multinode install? | 16:33 |
| *** banix has quit IRC | 16:34 | |
| *** banix_ is now known as banix | 16:34 | |
| inc0 | gmmaha, it does, not the ansible version check tho | 16:34 |
| inc0 | there are 2 tasks there, just pointing out it needs to behave like first | 16:34 |
| inc0 | vhosakot, and add when ubuntu | 16:34 |
| gmmaha | inc0: aah ok.. understand what you mean | 16:34 |
| inc0 | there are tools in ansible to make this if | 16:34 |
| vhosakot | inc0: if the inventory ansible/inventory/multinode is used, precehcks does run on every node.. is this correct ? | 16:37 |
| *** achanda has quit IRC | 16:37 | |
| gmmaha | vhosakot: i have seen it run on all hosts.. | 16:38 |
| vhosakot | gmmaha: cool.. | 16:38 |
| vhosakot | gmmaha: I will file the bug, could you update kernel and retry | 16:40 |
| *** banix has quit IRC | 16:41 | |
| gmmaha | vhosakot: doing that right now | 16:41 |
| vhosakot | cool | 16:41 |
| sbezverk | is it expected that nova_libvirt runs as root? | 16:42 |
| inc0 | sbezverk, well, nova_libvirt uses kernel stuff | 16:44 |
| inc0 | so it has to have more priviledges than normal | 16:44 |
| manjeets | inc0: I launched multiple instances which was working fine yesterday and i just exited yesterday re sourced my open rc | 16:46 |
| manjeets | today when i launch instances it says image could not be found | 16:46 |
| sbezverk | inc0 make sense, it is just I was under impresssion that no containers shold be running as root.. as a result when one tries to access libvirtd.log located on heka, the access gets denied.. | 16:47 |
| sbezverk | and for just one case to access logs, one need to use different command | 16:50 |
| *** unicell1 has quit IRC | 16:50 | |
| vhosakot | sbezverk: I have seen that... you need to do docker exec -u root heka cat /var/log/kolla/libvirt/libvirtd.log | 16:54 |
| *** Marga_ has joined #kolla | 16:55 | |
| vhosakot | sbezverk: neutron also uses rootwrap... not all services can be started as non-root I think... | 16:55 |
| inc0 | vhosakot, we do rootwrap with underprivileged sudo | 16:55 |
| vhosakot | ah ok | 16:55 |
| vhosakot | over-prevelieged non-root ? | 16:56 |
| inc0 | thats one way to put iy | 16:56 |
| vhosakot | yep.. cool | 16:56 |
| sbezverk | vhosakot I know about the command, it is just confusing for people who hopefully will use kolla.. | 16:58 |
| vhosakot | sbezverk: ah ok | 16:59 |
| inc0 | good call sbezverk | 16:59 |
| inc0 | can you access this file via /var/lib/docker/volumes/heka/_data/var/log/kolla/libvirt/libvirtd.log? | 16:59 |
| inc0 | path written from memory, may differ in real world | 17:00 |
| sbezverk | inc0 ls /var/lib/docker/volumes/heka/_data/ | 17:06 |
| sbezverk | logstreamer/ sandbox_preservation/ | 17:06 |
| sbezverk | inc0 in neither folders I could fine libvirt related log, but for other components yes.. | 17:06 |
| *** elemoine has quit IRC | 17:07 | |
| *** asalkeld has quit IRC | 17:07 | |
| *** thomasem has quit IRC | 17:07 | |
| *** patchbot has quit IRC | 17:07 | |
| *** dmsimard has quit IRC | 17:07 | |
| *** akwasnie has quit IRC | 17:07 | |
| *** tpeoples has quit IRC | 17:07 | |
| *** jmccarthy has quit IRC | 17:07 | |
| *** shakamunyi has quit IRC | 17:07 | |
| *** elemoine_ has quit IRC | 17:07 | |
| *** cloudnull has quit IRC | 17:07 | |
| *** smekel_ has quit IRC | 17:07 | |
| vhosakot | I need root access on host to view any file in /var/lib/docker/... I use sudo | 17:07 |
| *** sdake_ has joined #kolla | 17:07 | |
| inc0 | yeah I guess you might | 17:07 |
| inc0 | in any case, thats how I read logs;) | 17:07 |
| vhosakot | ah ok | 17:07 |
| manjeets | if i do docker attach container does it directs to live logs ? | 17:07 |
| *** patchbot` has joined #kolla | 17:07 | |
| *** elemoine- has joined #kolla | 17:07 | |
| *** sdake has quit IRC | 17:07 | |
| *** barra204 has joined #kolla | 17:07 | |
| *** patchbot` is now known as patchbot | 17:07 | |
| *** Marga_ has quit IRC | 17:07 | |
| *** dmsimard1 has joined #kolla | 17:07 | |
| *** Marga_ has joined #kolla | 17:07 | |
| *** thomasem_ has joined #kolla | 17:07 | |
| *** cloudnull has joined #kolla | 17:08 | |
| *** jmccarthy has joined #kolla | 17:08 | |
| vhosakot | sbezverk: I ahve it at ./volumes/kolla_logs/_data/libvirt/libvirtd.log | 17:09 |
| vhosakot | in /var/lib/docker | 17:09 |
| *** rajathagasthya has joined #kolla | 17:10 | |
| sbezverk | vhosakot again my opint was more for people usability than for hackish way of accessing info ;-) | 17:12 |
| *** unicell has joined #kolla | 17:13 | |
| *** unicell has quit IRC | 17:13 | |
| sdake_ | did we get gmmaha fixed up? | 17:14 |
| *** unicell has joined #kolla | 17:14 | |
| vhosakot | sbezverk: central_logging puts all the kolla logs in /var/lib/docker/volumes/kolla_logs/_data/ on the host.... why do you say it is hack-ish ? | 17:14 |
| *** elemoine_ has joined #kolla | 17:14 | |
| sdake_ | it should be libvirt, not _data | 17:14 |
| *** smekel_ has joined #kolla | 17:15 | |
| sdake_ | and it should bein the hkea container | 17:15 |
| inc0 | sdake_, it's how you access logs from outside of containers | 17:15 |
| inc0 | which tbh I do | 17:15 |
| gmmaha | sdake_: got the machines fixed up and deploying now | 17:15 |
| inc0 | gmmaha, fingers crossed | 17:15 |
| sdake_ | inc0 i ggot it | 17:15 |
| gmmaha | flushed all in /var/lib/docker and verified they are all running 4.2.0 | 17:15 |
| sdake_ | my point is we need to be able to access from inside continers :) | 17:15 |
| gmmaha | inc0: indeed.. | 17:15 |
| sdake_ | gmmaha i hope you rm -rff /var/lib/docker | 17:16 |
| sdake_ | rather then just flush the files | 17:16 |
| gmmaha | sdake_: yup.. | 17:16 |
| gmmaha | did that | 17:16 |
| gmmaha | stop daemon, rm -rf .. start daemon | 17:16 |
| rajathagasthya | Hi! Trying to setup Kolla dev environment and getting "No guest IP was given to the Vagrant core NFS helper." Has anyone seen this before? I did vagrant up --provider=virtualbox | 17:16 |
| sdake_ | who knows what your old kernel did to the filesystem | 17:16 |
| sdake_ | rajathagasthya pbourke is your man | 17:16 |
| vhosakot | gmmaha: dpkg -l | grep lts | 17:16 |
| sdake_ | rajathagasthya your other man is mandre | 17:16 |
| sbezverk | vhosakot I think there should be consistent way of accessing logs, regardless under what account container runs | 17:16 |
| sdake_ | sbezverk there is - its called the heka container | 17:17 |
| sbezverk | sdake I know but try to access libvirtd.log there | 17:17 |
| gmmaha | vhosakot: ii libsigsegv2:amd64 2.10-2 amd64 Library for handling page faults in a portable way | 17:17 |
| gmmaha | ii linux-image-generic-lts-wily 4.2.0.34.27 amd64 Generic Linux kernel image | 17:17 |
| sdake_ | ye slibvirt needs to log to heka | 17:17 |
| sdake_ | i thought someon ehad fixed thix already? | 17:17 |
| rajathagasthya | Thanks, sdake_. | 17:17 |
| inc0 | sdake_, it's post drop-root stuff | 17:18 |
| vhosakot | sdake_: I did it | 17:18 |
| vhosakot | gmmaha: the outpit looks good :) | 17:18 |
| inc0 | there are libvirt logs but with root privileges | 17:18 |
| sbezverk | sdake it does but it has different permissions | 17:18 |
| gmmaha | vhosakot: :) fingers crossed now | 17:18 |
| sdake_ | make them 644? | 17:18 |
| gmmaha | but the msot bizzare is how on earth did those machines lose that | 17:18 |
| gmmaha | aah failed iwth stupid IP not resolve issue.. | 17:18 |
| * gmmaha goes to fix /etc/hosts | 17:19 | |
| sbezverk | sdake the problem libvirt folder has right permissions | 17:19 |
| manjeets | sdake inc0 when i spin bunch of vm's some of them are getting two ips | 17:19 |
| inc0 | gmmaha, it's not stupid if you would fail later;) | 17:19 |
| inc0 | also remember to sync up ntp | 17:19 |
| sbezverk | but since libvirtd process runs as root and not part of kolla group | 17:19 |
| *** tpeoples has joined #kolla | 17:19 | |
| inc0 | manjeets, welcome to neutron;) | 17:19 |
| vhosakot | logs can be made 644 in heka then... sbezverk, would this work ? | 17:19 |
| sbezverk | it creates logs with 600 | 17:19 |
| gmmaha | inc0: :) | 17:19 |
| *** kproskurin has quit IRC | 17:19 | |
| manjeets | inc0 i guess its not neutron | 17:19 |
| manjeets | its rabbitmq | 17:19 |
| *** kproskurin has joined #kolla | 17:19 | |
| sbezverk | vhosakot please read my previous post | 17:20 |
| inc0 | well, it's neutron using rabbitmq | 17:20 |
| sdake_ | sbezverk clearly libvirt could be fixed to write its logs and 644 | 17:20 |
| sdake_ | and/as | 17:20 |
| inc0 | let me rephrase, welcome to wonderful world of openstack maintenance, here is your rabbitmq hole, jump head first and let us know what you land on | 17:20 |
| sdake_ | touch /var/lib/libvirt/libvirt.log | 17:20 |
| sdake_ | touch /var/lib/kolla/libvirt/libvirt.log | 17:20 |
| sdake_ | chmod 644 /var/lib/kolla/libvirt/libvirt.log | 17:21 |
| sdake_ | before libvirtd startup | 17:21 |
| sbezverk | sdake ok good idea I filed a bug for this issue | 17:21 |
| *** dmsimard1 is now known as dmsimard | 17:21 | |
| sbezverk | if you do not mind I will use this solution then.. | 17:21 |
| sdake_ | sbezverk where is bug - do o uwant to fix or shall i | 17:21 |
| sdake_ | wfm, add me as a coauthor if it works | 17:21 |
| sdake_ | Co-Authored-By: Steven Dake <stdake@cisco.com> | 17:22 |
| sdake_ | add at end of commit message | 17:22 |
| sbezverk | sdake sure thing | 17:22 |
| *** akwasnie has joined #kolla | 17:22 | |
| sdake_ | we always give credit to peopel that come up with fixes | 17:22 |
| inc0 | sdake_, getting his stackalytics up;) | 17:22 |
| sdake_ | (unless they dont want it) | 17:22 |
| sdake_ | nah its ust polite behavior inc0 | 17:22 |
| sdake_ | iv'e asked you every time we worke don something joint | 17:22 |
| vhosakot | sbezverk: yep.. so, the permission would be made 644... instead of this ? that is good... it will be consistent with other logs | 17:23 |
| sdake_ | you said no :) | 17:23 |
| inc0 | I know, I jest | 17:23 |
| vhosakot | $ docker exec heka ls -lrt /var/log/kolla/libvirt/libvirtd.log | 17:23 |
| vhosakot | -rw------- 1 root kolla 1584 Mar 28 23:58 /var/log/kolla/libvirt/libvirtd.log | 17:23 |
| inc0 | yup, that's pretty restrictive | 17:23 |
| vhosakot | inc0: sbezverk yep agreed | 17:23 |
| *** asalkeld has joined #kolla | 17:23 | |
| *** kproskurin has quit IRC | 17:25 | |
| *** vishwanathj has quit IRC | 17:28 | |
| gmmaha | vhosakot: nova_compute will go on reboot when libvirtd is runnign on host, right? | 17:28 |
| vhosakot | gmmaha: stop libvirt on host | 17:28 |
| gmmaha | vhosakot: yeah | 17:28 |
| gmmaha | going to | 17:28 |
| *** vishwanathj has joined #kolla | 17:28 | |
| * gmmaha hates ubuntu now... a lottttt | 17:28 | |
| vhosakot | i do this | 17:29 |
| vhosakot | sudo service libvirt-bin stop | 17:29 |
| vhosakot | sudo update-rc.d libvirt-bin disable | 17:29 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Fixing permission for libvirtd.log https://review.openstack.org/298887 | 17:30 |
| inc0 | gmmaha, also make sure to remove ovs from host | 17:30 |
| gmmaha | vhosakot: the second command always fails on ubuntu | 17:30 |
| gmmaha | inc0: yeah that got kicked out | 17:31 |
| manjeets | what is solution to remove split brain issue of rabbitmq | 17:31 |
| inc0 | sbezverk, so it worked? | 17:31 |
| vhosakot | gmmaha: yep, it fails it that binary does not exist.. np | 17:31 |
| gmmaha | vhosakot: seen this before? http://paste.openstack.org/show/492328/ | 17:32 |
| gmmaha | will start disable libvirtd on all machines now | 17:32 |
| vhosakot | share /run mount | 17:32 |
| *** inc0 has quit IRC | 17:32 | |
| *** sdake has joined #kolla | 17:33 | |
| gmmaha | ohh yeah | 17:33 |
| *** inc0 has joined #kolla | 17:33 | |
| *** achanda has joined #kolla | 17:33 | |
| *** Marga__ has joined #kolla | 17:34 | |
| *** sdake_ has quit IRC | 17:34 | |
| sbezverk | inc0 when tricked manually yes, now testing with normal deployment | 17:35 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr ansible role https://review.openstack.org/298894 | 17:36 |
| *** Marga__ has quit IRC | 17:36 | |
| *** Marga_ has quit IRC | 17:37 | |
| vhosakot | sbezverk: so, "-u root" will not be needed to access the logs in heka now ? | 17:37 |
| *** Marga_ has joined #kolla | 17:37 | |
| *** dwalsh has joined #kolla | 17:38 | |
| *** Marga_ has joined #kolla | 17:38 | |
| sbezverk | vhosakot right | 17:38 |
| vhosakot | ah ok | 17:38 |
| *** Marga_ has quit IRC | 17:39 | |
| sbezverk | vhosakot to be consistent, I will check all other logs to see if any other logs require "root" to access | 17:39 |
| vhosakot | ok cool | 17:39 |
| *** Marga_ has joined #kolla | 17:40 | |
| vhosakot | gmmaha: so, there must be way to make that shared /run mount persist thru reboot on Ubuntu... will keep you posted | 17:40 |
| inc0 | fstab magics? | 17:40 |
| mlima | hello guys, this task failed TASK: [rabbitmq | fail msg="Hostname has to resolve to IP address of api_interface"] | 17:40 |
| gmmaha | mlima: check your /etc/hosts | 17:41 |
| gmmaha | most probably you ahve something like this | 17:41 |
| gmmaha | 127.0.1.1 <hostname> | 17:41 |
| mlima | yes | 17:41 |
| gmmaha | you want to remove that line | 17:41 |
| inc0 | also with ubntu 15.10 you can set this up in systemd stuff | 17:41 |
| mlima | ok | 17:41 |
| gmmaha | your machine should resolve to the IP that you are giving in your configuration | 17:41 |
| vhosakot | inc0: I checked.... "grep run /etc/fstab" is empty | 17:41 |
| mlima | ok | 17:41 |
| mlima | tahnk you gmmaha | 17:42 |
| gmmaha | mlima: and you dont need a reboot | 17:42 |
| mlima | :) | 17:42 |
| inc0 | vhosakot, well, maybe you need to add one;) | 17:42 |
| gmmaha | mlima: anytime :) | 17:42 |
| inc0 | gmmaha, mlima ha! my trap works, people walk into it | 17:42 |
| vhosakot | ah ok... | 17:42 |
| vhosakot | don't know if kolla should touch operator's fstab | 17:42 |
| inc0 | vhosakot, kolla should not, but we can ask operators to do it | 17:43 |
| mlima | inc0 you are bad :P | 17:43 |
| inc0 | same as we ask them to upgrade their kernel | 17:43 |
| vhosakot | yep | 17:43 |
| gmmaha | inc0: :) what trap? ; | 17:43 |
| inc0 | which is considerably more invasive;) | 17:43 |
| inc0 | gmmaha, host resolver check | 17:43 |
| gmmaha | inc0: aaah | 17:43 |
| gmmaha | :) | 17:43 |
| gmmaha | thats good.. | 17:43 |
| gmmaha | guessing ubuntu is the one writing stray crap like this | 17:44 |
| inc0 | yeah it does | 17:44 |
| inc0 | so life will be easyier with buntu 16.04 | 17:44 |
| gmmaha | inc0: i know.. | 17:45 |
| gmmaha | cnt wait! | 17:45 |
| gmmaha | vhosakot: inc0: now nova_compute is acting up http://paste.openstack.org/show/492333/ | 17:45 |
| gmmaha | i verified that its disabled | 17:45 |
| gmmaha | hehhe | 17:45 |
| gmmaha | kill me.. | 17:46 |
| gmmaha | kill me now!! | 17:46 |
| gmmaha | igonore it | 17:46 |
| gmmaha | vhosakot: inc0: i set libvirtd to be this 'sleep infinity' yesterday to test it :| | 17:46 |
| * gmmaha needs a gallon of coffee | 17:46 | |
| ntpttr | gmmaha: or one Viso | 17:47 |
| gmmaha | ntpttr: $23 for a bottle of drink.. | 17:47 |
| gmmaha | ?? | 17:47 |
| inc0 | so gmmaha after careful debugging I think error lies between chair and monitor:P | 17:48 |
| ntpttr | gmmaha: what? no, a viso is probably a couple of bucks.. | 17:48 |
| gmmaha | inc0: :D | 17:48 |
| gmmaha | http://drinkviso.com/ | 17:49 |
| ntpttr | gmmaha: "12 pack bottles" | 17:50 |
| gmmaha | ntpttr: aah :) | 17:50 |
| * gmmaha walks to get some cofee before he blabbers more | 17:50 | |
| sdake | gmmaha undo that | 17:51 |
| sdake | the sleep infinity | 17:51 |
| gmmaha | sdake: yeah | 17:51 |
| gmmaha | doing that now! | 17:51 |
| sdake | fwiw i thought early on your kernel was old | 17:52 |
| sdake | but someone verified it was valid | 17:52 |
| sdake | now i know. 4.20 or less is not valid for ubuntu ;) | 17:52 |
| gmmaha | ohh.. so things should work with 3.13 kernel? | 17:52 |
| gmmaha | sdake: aaah | 17:52 |
| gmmaha | :) | 17:52 |
| sdake | 3.15 is the miminimum i'd recommend | 17:52 |
| inc0 | yeah, 4.20 works, not sure how old it can get to keep working | 17:53 |
| inc0 | anyway, 16.04 will change a lot in our lives | 17:53 |
| *** gfidente has quit IRC | 17:53 | |
| vhosakot | 4.2 works for me as well... | 17:53 |
| inc0 | docker run by systemd, no more kernel upgrade requirement | 17:53 |
| vhosakot | gmmaha: can you undo the sleep infinity in Dockerfile and rebuild | 17:53 |
| inc0 | no more this mount shared crap | 17:53 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Fixing permission for libvirtd.log https://review.openstack.org/298887 | 17:54 |
| gmmaha | vhosakot: yeah.. doing that | 17:54 |
| sbezverk | sdake I got some progress about rabbitmq crash | 17:56 |
| *** sdake has quit IRC | 17:57 | |
| sbezverk | it looks like this file /var/lib/rabbitmq/.erlang.cookie is incorrectly generated when rabbitmq gets deployed | 17:57 |
| gmmaha | vhosakot: while i wait for the build & deploy.. what do you think about the change in horizon? | 17:58 |
| gmmaha | the _member_ | 17:58 |
| gmmaha | i believe it definitely has to go | 17:58 |
| vhosakot | gmmaha: I needed steps to reproduce that issue... I always use CLI and see no issue.... | 18:01 |
| vhosakot | can you pastebin the steps to uncover the bug please | 18:01 |
| gmmaha | vhosakot: install openstack using kolla, go to horizon UI, try to create a new project/user | 18:02 |
| gmmaha | if you use CLI it works fine.. | 18:02 |
| vhosakot | as admin in horizon ? | 18:02 |
| gmmaha | UI is what crashes as we are setting KEYSTONE_DEFAULT_ROLE= _member_ and horizon looks for it before it lets you do it | 18:02 |
| gmmaha | vhosakot: yeah | 18:02 |
| *** sdake has joined #kolla | 18:02 | |
| sbezverk | inc0 permission fix is working docker exec heka ls -al /var/log/kolla/libvirt/libvirtd.log | 18:04 |
| sbezverk | -rw-r--r--. 1 root kolla 1643 Mar 29 19:02 /var/log/kolla/libvirt/libvirtd.log | 18:04 |
| sbezverk | could you please review it if you have time | 18:04 |
| inc0 | yay | 18:04 |
| *** sdake has quit IRC | 18:08 | |
| *** sdake_ has joined #kolla | 18:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gate by adding MariaDB-shared dependnecy https://review.openstack.org/298479 | 18:08 |
| vhosakot | sbezverk: yep.... -rw-r--r-- looks nice! | 18:08 |
| sdake_ | sbezverk nice job! :) | 18:09 |
| sbezverk | oh well I just filed the bug, sdake came up with a trick.. | 18:10 |
| sbezverk | ;-) | 18:10 |
| gmmaha | vhosakot: sdake_: same issue with the new kernel.. http://paste.openstack.org/show/492339/ | 18:10 |
| vhosakot | ah | 18:10 |
| sbezverk | sdake rabbitmq issue is due to incorrectly generated .erlang.cookie | 18:11 |
| sdake_ | gmmaha did you reubild your host node | 18:11 |
| sdake_ | and set /run to shared | 18:11 |
| sdake_ | sbezverk do you know how to fix? | 18:11 |
| gmmaha | sdake_: aah didnt touch the hsot node at all. :| | 18:11 |
| sdake_ | can you file a bug plz | 18:11 |
| sdake_ | gmmaha you need to build with a proper kernel too | 18:11 |
| sdake_ | apologies if the documentation isn't clear | 18:12 |
| gmmaha | sdake_: aah the host has the right kernel | 18:12 |
| sdake_ | if you need changes for docs please submit them and they will be approved | 18:12 |
| sbezverk | sdake not yet, need to find a process that generates it.. | 18:12 |
| gmmaha | sdake_: i definitely will.. | 18:12 |
| sdake_ | use nuclear option on docker | 18:13 |
| sdake_ | and rebuild plz | 18:13 |
| sdake_ | just your host machine | 18:13 |
| sdake_ | you will have to redeploy your registry | 18:13 |
| sdake_ | i suspecty ou deployed your registry on an old kernel | 18:13 |
| gmmaha | sdake_: hmm possible.. | 18:13 |
| sdake_ | or rather its somethign id liek to rule out | 18:13 |
| sdake_ | because your the only person who has this problem | 18:14 |
| gmmaha | aah | 18:14 |
| ccesario | I 'm running Ubuntu 14.04 with linux-image-generic-lts-wily kernel without this problem.... | 18:15 |
| vhosakot | me too.. | 18:15 |
| gmmaha | ccesario: vhosakot: cool.. thanks.. yeah something went amuck on these machines | 18:16 |
| gmmaha | was doing just fine last week | 18:16 |
| ccesario | what did you change from last week until now ?!: ehehe | 18:17 |
| gmmaha | ccesario: nothing :( | 18:17 |
| gmmaha | thats the problem | 18:17 |
| gmmaha | worked friday, and monday its disaster | 18:18 |
| ccesario | O_o | 18:21 |
| *** Marga__ has joined #kolla | 18:21 | |
| mlima | sdake_, there is already some proposal or idea of how it will be implementing support for more backend for some services? | 18:22 |
| gmmaha | vhosakot: think its better to delete al the local imags on the host? | 18:22 |
| sdake_ | mlima you mean introduce more big tent services? | 18:22 |
| ccesario | gmmaha, well... try to do the steps that gurus said | 18:22 |
| mlima | cinder has many backends, but so far we only have the ceph | 18:22 |
| sdake_ | right plugins | 18:22 |
| gmmaha | ccesario: yeah thats what i am trying | 18:23 |
| mlima | oh | 18:23 |
| sdake_ | there are several plugin approache used by plugins | 18:23 |
| gmmaha | blew out the registry | 18:23 |
| gmmaha | brought a new one up | 18:23 |
| sdake_ | gmmaha make usre to use the nuclear option | 18:23 |
| sdake_ | mlima i expect plugins to take prime spot at our austin summit slots | 18:23 |
| gmmaha | sdake_: is that when i build the images? | 18:23 |
| sdake_ | gmmaha before deploying docker registry | 18:23 |
| sdake_ | stop docker, rm -rf /var/lib/docker, start docker, start registry, build images and push | 18:24 |
| gmmaha | sdake_: aah | 18:24 |
| gmmaha | duh.. i was searching for a command line option called nuclear | 18:24 |
| vhosakot | do those steps with master | 18:24 |
| gmmaha | vhosakot: yup.. doing that now.. thanks | 18:25 |
| *** Marga_ has quit IRC | 18:26 | |
| vhosakot | ccesario: we have ceph... sbezverk is doing lvm2 backend for cinder... what other backend drivers is needed please let us know | 18:26 |
| vhosakot | nfs gluster ? | 18:28 |
| sbezverk | sdake I think I know why rabbitmq was crashing at least in my case. rabbitmq_cluster_cookie: variable in passwords.yml was empty. | 18:29 |
| vhosakot | sbezverk: kolla-genpwd | 18:30 |
| sbezverk | vhosakot sorry I cannot overload my brain with generated passwords ;-) I like to cisco's most secured password ;-) | 18:33 |
| ccesario | vhosakot, I think that you was targeting mlima | 18:33 |
| ccesario | about ceph | 18:34 |
| vhosakot | ah... I wanted to message mlima :) | 18:34 |
| vhosakot | sbezverk: yep... that script generates all UUID4 values for passwords :) | 18:34 |
| sdake_ | sbezverk add a command line option to genpwd to take a pasword | 18:37 |
| sdake_ | we need that :) | 18:37 |
| sdake_ | also a command line optoin to limit length of autogeneated passwords | 18:38 |
| sdake_ | inc0 are you unable to use webex inside intel? | 18:38 |
| inc0 | never tried besides calling in | 18:39 |
| inc0 | and I call in with phone | 18:39 |
| sdake_ | inc0 can we try real quick? | 18:39 |
| sdake_ | sbezverk what would be my personal meeting room? | 18:39 |
| sdake_ | i cant remember the details | 18:40 |
| sdake_ | inc0 webex is needed to debug this mariadb lights out failure during bootstrapping | 18:40 |
| *** achanda has quit IRC | 18:40 | |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1563500 | 18:41 |
| openstack | Launchpad bug 1563500 in kolla "prechecks must check LTS wily kernel version 4.2 on Ubuntu 14.04" [Undecided,In progress] - Assigned to Vikram Hosakote (vhosakot) | 18:41 |
| vhosakot | I will open another bug to make precehcks check for empty passwords | 18:41 |
| sdake_ | vhosakot osunds good | 18:42 |
| sdake_ | sbezverk cn you throw me a bone and tell me what my personal meeting room is | 18:45 |
| sdake_ | or vhosakot | 18:45 |
| sdake_ | inc0 it does require windows or mac though sa a client | 18:45 |
| inc0 | sdake_, so infra manjeets has is not in intel infra | 18:46 |
| sdake_ | yes but he can get to it from his workstation | 18:46 |
| mlima | vhosakot, some vendor drivers are available in cinder too | 18:46 |
| sdake_ | therea re like 50 vendor drivers for cinder | 18:46 |
| vhosakot | my room is at https://cisco.webex.com/join/vhosakot.... i use to host meetings | 18:46 |
| sdake_ | and 30 or 40 for neutron | 18:46 |
| sdake_ | we need to sort out how to deal with that | 18:46 |
| vhosakot | mlima: you mean EMC, netapp ? | 18:46 |
| mlima | yes | 18:46 |
| mlima | i don't know, but can be interesting supports them vhosakot | 18:48 |
| *** achanda has joined #kolla | 18:49 | |
| mlima | but I think that once defined how this can be done will depend on the interest of those who want to implement this | 18:49 |
| vhosakot | https://bugs.launchpad.net/kolla/+bug/1563506 | 18:51 |
| openstack | Launchpad bug 1563506 in kolla "prechecks must check empty passwords in /etc/kolla/passwords.yml" [Undecided,In progress] - Assigned to Vikram Hosakote (vhosakot) | 18:51 |
| *** Marga__ has quit IRC | 18:52 | |
| *** Marga_ has joined #kolla | 18:53 | |
| *** dwalsh has quit IRC | 18:58 | |
| *** dwalsh has joined #kolla | 18:59 | |
| *** Marga__ has joined #kolla | 19:00 | |
| sdake_ | yay I got a platinum amex card from work :) | 19:00 |
| *** sdake_ is now known as sdake | 19:01 | |
| *** Marga_ has quit IRC | 19:03 | |
| *** vhosakot has quit IRC | 19:03 | |
| inc0 | sdake then we drink more old port in Austin? | 19:03 |
| sdake | i think the days of 350$ bottle of port are long gone ;) | 19:04 |
| sdake | atleast until vestmas ;) | 19:04 |
| *** vhosakot has joined #kolla | 19:04 | |
| sdake | but drinks not on the first page are a possiblity ;) | 19:04 |
| sdake | i am at the w, where are you staing? | 19:05 |
| gmmaha | sdake: vhosakot: same error even after all the cleanup | 19:06 |
| inc0 | sdake, I'm in hilton | 19:07 |
| inc0 | but Austin is full of nice places | 19:07 |
| gmmaha | i give up and am officially throwing in the towel | 19:09 |
| *** akscram has quit IRC | 19:14 | |
| *** akscram has joined #kolla | 19:15 | |
| sdake | https://review.openstack.org/298943 | 19:18 |
| sdake | gmmaha no giving up ;) | 19:18 |
| sdake | gmmaha my recommendation is to take a step back and relax for a day, and we tackle it tomorrow with a live debug session | 19:19 |
| sdake | you and i will be fresh instead of burning the last 20 hours trying to get you going again | 19:19 |
| sdake | vhosakot can you totallly rebuild your containers and see if you can duplicate gmmaha's problem in the meantime? | 19:20 |
| sdake | I'd recommend tagging them as 2.0.1 when pushing to the registry | 19:20 |
| sdake | in case upstream libvirt is busted | 19:20 |
| sdake | then put openstack_release in globals.yml | 19:20 |
| *** salv-orlando has joined #kolla | 19:22 | |
| *** salv-orl_ has quit IRC | 19:24 | |
| vhosakot | sdake: ok... will keep you guys posted | 19:24 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Starting heka bootstrap container failed in deployment https://review.openstack.org/298944 | 19:24 |
| *** huikang has joined #kolla | 19:27 | |
| *** shardy is now known as shardy_afk | 19:28 | |
| huikang | Good afternoon | 19:28 |
| *** salv-orl_ has joined #kolla | 19:32 | |
| *** salv-orlando has quit IRC | 19:35 | |
| vhosakot | huikang: afternoon | 19:36 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Pass extra arguments through kolla-ansible to ansible-playbook https://review.openstack.org/297394 | 19:38 |
| openstackgerrit | Merged openstack/kolla: Restrict the rabbitmq only start on rabbitmq group https://review.openstack.org/297929 | 19:47 |
| *** sdake_ has joined #kolla | 19:55 | |
| sdake_ | hey huikang | 19:57 |
| sdake_ | good to see you on irc! | 19:57 |
| *** sdake has quit IRC | 19:58 | |
| *** huikang has quit IRC | 20:00 | |
| *** sdake_ is now known as sdake | 20:02 | |
| *** salv-orlando has joined #kolla | 20:03 | |
| gmmaha | sdake: :) | 20:04 |
| gmmaha | alright.. that seems fair. | 20:04 |
| gmmaha | vhosakot: thanks | 20:04 |
| *** salv-orl_ has quit IRC | 20:05 | |
| *** rhallisey has quit IRC | 20:08 | |
| *** rajathagasthya has quit IRC | 20:10 | |
| *** Marga__ has quit IRC | 20:10 | |
| *** Marga_ has joined #kolla | 20:12 | |
| sbezverk | sdake got confirmed crashing rabbitmq was due to missing cookie password.. | 20:17 |
| sdake | in tc meeting, but the cookie password shuld not be missing | 20:17 |
| *** achanda has quit IRC | 20:21 | |
| manjeets | can we dedicate each rabbit mq to speicfic network node | 20:23 |
| manjeets | sdake inc0 | 20:23 |
| *** dwalsh has quit IRC | 20:25 | |
| vhosakot | manjeets: rabbitmq must be on each controller | 20:27 |
| *** vhosakot has quit IRC | 20:32 | |
| manjeets | some of vms are getting 2 or 3 ips | 20:33 |
| *** pbourke-home has joined #kolla | 20:33 | |
| *** rhallisey has joined #kolla | 20:41 | |
| *** pbourke-home has quit IRC | 20:44 | |
| *** huikang has joined #kolla | 20:45 | |
| *** lifeless has joined #kolla | 20:47 | |
| lifeless | mm, I thought I was here. shrug :) | 20:47 |
| lifeless | sdake: so | 20:47 |
| *** ayoung has quit IRC | 20:54 | |
| sdake | well that was fun | 20:57 |
| sdake | ok lifeless | 20:57 |
| sdake | stateless | 20:57 |
| *** shardy_afk has quit IRC | 20:57 | |
| sdake | we aren't totatlly stateless | 20:57 |
| sdake | docker-enine contains our state | 20:57 |
| sdake | but the tools tht interact with docker-engine are idempotent and stateless | 20:58 |
| sdake | if docker-engine is missing a container - a kolla deploy will fix it up | 20:58 |
| sdake | then the next time it goes to deploy, if the state is already there, nothing is done | 20:58 |
| sdake | we also have an upgrade and reconfigure action | 20:58 |
| sdake | which do as they sound | 20:59 |
| sdake | they upgrade from one version ofopenstack to the next or apply completely custom configuration via ini file merging | 20:59 |
| sdake | so in some regard, our config files contain some state | 20:59 |
| sdake | iff there is custom state | 20:59 |
| sdake | and the jinja2 defaults for ansible contain some state | 20:59 |
| sdake | that get applied on the fly | 21:00 |
| sdake | but overall ansible is completely statleess and daemon free (minus sshd off course) | 21:00 |
| sdake | our deployment tool is a simple shell wrapper around ansible | 21:00 |
| sdake | second for source: | 21:00 |
| sdake | https://github.com/openstack/kolla/blob/master/tools/kolla-ansible#L39 | 21:01 |
| lifeless | thanks | 21:02 |
| lifeless | so | 21:03 |
| lifeless | much of nova is stateless too, under that definition | 21:03 |
| lifeless | in that its state lives in a db | 21:03 |
| lifeless | and so does kollas | 21:03 |
| lifeless | I wonder if some refactoring around the concept of stored state | 21:04 |
| lifeless | might make the discussion about what deliverables upgrade should be applicable to clearer | 21:04 |
| sdake | lifeless intereting idea | 21:20 |
| sdake | sorry 3 meetings atm ;) | 21:20 |
| *** vhosakot has joined #kolla | 21:21 | |
| *** achanda has joined #kolla | 21:21 | |
| *** inc0 has quit IRC | 21:26 | |
| *** banix has joined #kolla | 21:27 | |
| *** achanda has quit IRC | 21:27 | |
| vhosakot | huikang: ping | 21:32 |
| vhosakot | had some questions about https://review.openstack.org/#/c/297394/2/tools/kolla-ansible | 21:32 |
| patchbot | vhosakot: patch 297394 - kolla - Pass extra arguments through kolla-ansible to ansi... | 21:32 |
| *** ccesario_ has joined #kolla | 21:33 | |
| vhosakot | huikang: I think we both mean the same... YAML is a file at the end of the day.. :) | 21:34 |
| *** sdake_ has joined #kolla | 21:35 | |
| *** jeblair has joined #kolla | 21:35 | |
| *** banix has quit IRC | 21:35 | |
| *** ChanServ sets mode: +o jeblair | 21:37 | |
| *** ccesario_ has quit IRC | 21:37 | |
| *** jeblair changes topic to "We have moved to #openstack-kolla, please /part and then type /join #openstack-kolla to get to us" | 21:38 | |
| sdake_ | lifeless any hance you can comment in the review about your irc comments | 21:38 |
| *** sdake has quit IRC | 21:38 | |
| jeblair | Hi all, I'm about to set this channel up to automatically redirect to #openstack-kolla | 21:38 |
| sdake_ | i think they are good and would appreciate them in the review itself so i can coordinate everyone's comments | 21:38 |
| *** ChanServ has joined #kolla | 21:39 | |
| *** services. sets mode: +o ChanServ | 21:39 | |
| *** ChanServ sets mode: +imsf #openstack-kolla | 21:39 | |
| jeblair | Okay, the redirect is set up now. anyone who tries to join this channel will be automatically redirected to #openstack-kolla | 21:40 |
| jeblair | Everyone here should leave this channel and join that one now | 21:41 |
| *** jtriley has quit IRC | 21:43 | |
| *** sbezverk has left #kolla | 21:43 | |
| *** sdake_ has left #kolla | 21:47 | |
| *** huikang has quit IRC | 21:49 | |
| jeblair | ccesario_: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | gbraad: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | lifeless: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | pbourke: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | The_Ball: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | ajafo: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | cfarquhar: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | gmmaha: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | logan-: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | peterjenkins: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | thomasem_: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | akscram: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | clayton: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | haplo37: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | m3m0: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | ph03n1x: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | tpeoples: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | akwasnie: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | cloudnull: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | harmw: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | mandre: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | rhallisey: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | tuvenen: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | akwasnie_: This channelis being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | daneyon: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | huikang: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | manjeets: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | ruhe: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | unicell: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | alisonh: This channel is being moved. Please /part and /join #openstack-kolla | 21:52 |
| jeblair | dgonzalez: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | intr1nsic: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | Marga_: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | salv-orlando: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | vhosakot: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | asalkeld: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | dims: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | mdnadeem_: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | sbezverk: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | vincent_vdk: This channel is being moved. Please /part and /join #openstack-kolla | 21:53 |
| jeblair | asalkeld_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | diogogmt: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | jgriffith: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | mgoddard: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | sdake: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | vishwanathj: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | banix: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | dmsimard: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | jmccarthy: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | mkoderer: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | sdake_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | yuanying: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | barra204: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | dougs: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | jpeeler: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | mkoderer__: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | sean-k-m1: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | zeus: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | bigjools: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | DuncanT: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | jtriley: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | mlima: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | serverascode: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | zigo: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | blahRus: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | elemoine-: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | kbyrne: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | nihilifer: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | SiRiuS_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | brad[] : This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | elemoine_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | Kennan: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | ntpttr: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | slagle: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | bradjones: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | elmiko: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | kjejlly_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | Slower_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | cbaesema: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | esimone: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | kklimonda: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | smekel_: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | ccesario: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | g3ek: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| jeblair | larsks: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| *** serverascode has left #kolla | 21:54 | |
| jeblair | patchbot: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| *** zigo has left #kolla | 21:54 | |
| jeblair | stvnoyes: This channel is being moved. Please /part and /join #openstack-kolla | 21:54 |
| *** DuncanT has left #kolla | 21:55 | |
| *** SiRiuS_ has left #kolla | 21:55 | |
| *** rhallisey has left #kolla | 21:55 | |
| *** ntpttr has left #kolla | 21:55 | |
| *** vhosakot has left #kolla | 21:55 | |
| *** jpeeler has left #kolla | 21:55 | |
| *** elmiko has left #kolla | 21:55 | |
| *** Marga_ has left #kolla | 21:55 | |
| *** dgonzalez has left #kolla | 21:58 | |
| *** daneyon has left #kolla | 21:58 | |
| *** lifeless has left #kolla | 21:59 | |
| *** manjeets has left #kolla | 22:01 | |
| *** blahRus has left #kolla | 22:04 | |
| *** haplo37 has quit IRC | 22:07 | |
| *** dims has left #kolla | 22:13 | |
| *** vishwanathj has left #kolla | 22:23 | |
| *** larsks has left #kolla | 22:41 | |
| *** unicell has quit IRC | 22:43 | |
| *** salv-orlando has left #kolla | 22:43 | |
| *** diogogmt has quit IRC | 22:48 | |
| *** clayton has left #kolla | 22:58 | |
| *** asalkeld_ has left #kolla | 23:00 | |
| *** gmmaha has left #kolla | 23:16 | |
| *** kklimonda has left #kolla | 23:28 | |
| *** dmsimard has left #kolla | 23:29 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!