| *** alisonh has joined #kolla | 00:02 | |
| *** JoseMello has quit IRC | 00:08 | |
| *** yuanying has quit IRC | 00:09 | |
| *** yuanying has joined #kolla | 00:16 | |
| *** jasonsb has joined #kolla | 00:16 | |
| *** tzn has joined #kolla | 00:19 | |
| *** yuanying has quit IRC | 00:19 | |
| *** yuanying has joined #kolla | 00:22 | |
| *** achanda has joined #kolla | 00:27 | |
| *** tzn has quit IRC | 00:35 | |
| *** sdake has joined #kolla | 00:35 | |
| *** ssurana has quit IRC | 00:37 | |
| *** farid has quit IRC | 00:37 | |
| *** sdake_ has joined #kolla | 00:50 | |
| *** sdake has quit IRC | 00:54 | |
| *** Allen_Gao has joined #kolla | 00:54 | |
| SamYaple | inc0: here | 00:55 |
|---|---|---|
| SamYaple | inc0: you still around? | 00:55 |
| inc0 | yeah | 00:55 |
| inc0 | what was that about? | 00:55 |
| SamYaple | ok so you want bad news first? | 00:56 |
| inc0 | bring it on | 00:56 |
| SamYaple | asalkeld: yes things changed, yes ive seen that before, ill tell you in a second | 00:56 |
| SamYaple | ok inc0 | 00:56 |
| SamYaple | successfully removed and used a _new_ libvirt image, no loss of qemu process | 00:56 |
| SamYaple | issue is the new libvirt contaienr did _not_ have a record of the running qemu instance | 00:57 |
| SamYaple | the reason for this is we are missing a file mount (i beielve) with needed info | 00:57 |
| SamYaple | thats the good news I suppose | 00:57 |
| SamYaple | bad news is it _still_ seems well have downtime with this issue | 00:58 |
| SamYaple | but we won't have downtime moving foward | 00:58 |
| inc0 | so, we might be able to reconnect to qemu | 00:58 |
| inc0 | if we find out exactly what libvirt does to reconnect | 00:58 |
| SamYaple | there is a reconnect command | 00:58 |
| SamYaple | virsh qemu-reconnect <pid> | 00:59 |
| SamYaple | but thats not the whole picture | 00:59 |
| *** Kennan is now known as Kennan_Vacation | 00:59 | |
| SamYaple | im working on identifying the issue for loss of info in libvirt | 00:59 |
| SamYaple | but i think it will still require a hard downtime to fix | 00:59 |
| SamYaple | not positive though | 00:59 |
| inc0 | I'd assume libvirt has some qemu-pid file somewhere | 00:59 |
| SamYaple | virsh qemu-attach <pid> *** | 00:59 |
| SamYaple | i know the pid | 01:00 |
| SamYaple | it didnt reattach | 01:00 |
| inc0 | and it disapears with container | 01:00 |
| SamYaple | like I said, its not wanting to reattach | 01:00 |
| inc0 | you know, but new libvirt doesn't | 01:00 |
| SamYaple | yes it does | 01:00 |
| SamYaple | i ran that command | 01:00 |
| SamYaple | it errors | 01:00 |
| inc0 | ah, I see | 01:00 |
| inc0 | any helpful error? | 01:00 |
| *** jtriley has joined #kolla | 01:00 | |
| inc0 | I'll ask around if anyone in my glorious company had anything to do with libvirt, maybe they could help | 01:01 |
| SamYaple | asalkeld: that issue is a keystone v2/v3 issue. i remember it from when i rwote the module | 01:01 |
| SamYaple | inc0: kinda. im still working through it | 01:01 |
| SamYaple | its possible, thats all ill say | 01:02 |
| inc0 | well, it's hopeful | 01:02 |
| SamYaple | its definetly happening long term | 01:02 |
| inc0 | does it attach if we move libvirt outside? | 01:02 |
| SamYaple | but since we missed needed info we may need a hard downtime to fix | 01:02 |
| SamYaple | nah but thats another issue | 01:02 |
| SamYaple | since it doenst have access to nova mountpoint | 01:02 |
| inc0 | a yeah | 01:02 |
| SamYaple | it would otherwise | 01:03 |
| inc0 | well it kinda does if we navigate through /var/lib/docker | 01:03 |
| SamYaple | but then again we wouldnt need to restart libvirt ;) | 01:03 |
| inc0 | apart from possible fs problems | 01:03 |
| SamYaple | it would work apart from that yes | 01:03 |
| inc0 | but we're not doing it, too ugly even for me | 01:03 |
| inc0 | and can very well fail horribly | 01:04 |
| SamYaple | thats a preq for libvirt on the host just so you know | 01:04 |
| SamYaple | bind mounting host /var/lib/nova and /var/lib/libvirt | 01:04 |
| inc0 | another issue | 01:04 |
| inc0 | to discuss over a beer | 01:04 |
| SamYaple | eh you say ugly I say useful | 01:05 |
| SamYaple | thats how yaodu did it | 01:05 |
| SamYaple | im not saying its better than what we have, but its not _bad_ | 01:05 |
| inc0 | I'm saying navigating through /var/lib/docker is ugly, bindmountig is ok | 01:05 |
| SamYaple | OHHH | 01:05 |
| SamYaple | yea i agree | 01:05 |
| SamYaple | no /var/lib/docker | 01:05 |
| inc0 | so we need to figure out how exactly libvirt attach works | 01:06 |
| inc0 | maybe there is indeed missing file, but one we can scoop out before tearing down old container | 01:06 |
| SamYaple | thats my thinking | 01:06 |
| SamYaple | im searching for said files | 01:06 |
| SamYaple | no luck | 01:06 |
| inc0 | technically containers are immutable, and since it's reattaching it doesn't keep anything in memory | 01:06 |
| SamYaple | all i know about are files in /var/lib/libvirt and /var/lib/nova | 01:06 |
| SamYaple | its likely a db somewhere i dont know about | 01:07 |
| inc0 | I'll look for someone who knows libvirt | 01:07 |
| SamYaple | i was at dinner right after discovery above | 01:08 |
| SamYaple | so im checking again | 01:08 |
| asalkeld | thanks SamYaple | 01:09 |
| *** neilus has joined #kolla | 01:10 | |
| SamYaple | asalkeld: my goal is to switch those modules over to the shade modules before mitaka | 01:10 |
| SamYaple | but thats just playbook changes | 01:10 |
| SamYaple | you can do that in mesos right now | 01:10 |
| asalkeld | ok | 01:10 |
| asalkeld | i think longer term we want to just not use ansible for this stuff | 01:11 |
| *** yuanying has quit IRC | 01:11 | |
| *** yuanying has joined #kolla | 01:12 | |
| SamYaple | asalkeld: yea honestly mesos is way more kolla-ansibly than i thought it would be | 01:12 |
| *** iNeilus has quit IRC | 01:12 | |
| SamYaple | frankly you guys could just call the openstackclient inside the kolla-ansible container and be done with it | 01:13 |
| asalkeld | yip | 01:15 |
| *** CheKoLyN has quit IRC | 01:19 | |
| *** yuanying has quit IRC | 01:21 | |
| *** Jeffrey4l has joined #kolla | 01:22 | |
| *** yuanying has joined #kolla | 01:23 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 01:28 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap https://review.openstack.org/275504 | 01:28 |
| *** sdake_ has quit IRC | 01:33 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 01:37 |
| *** Jeffrey4l has quit IRC | 01:39 | |
| *** tfukushima has joined #kolla | 01:40 | |
| *** alisonh has quit IRC | 01:47 | |
| *** alisonh has joined #kolla | 01:51 | |
| *** farid has joined #kolla | 02:03 | |
| Allen_Gao | SamYaple: when attaching qemu process to libvirt, * - The '-name' and '-uuid' arguments should have been set (not mandatory, but strongly recommended) | 02:10 |
| Allen_Gao | SamYaple: And FYI, It must have been started with a monitor socket using the UNIX domain socket protocol. | 02:10 |
| Allen_Gao | SamYaple: And ensure that No device hotplug/unplug, or other configuration changes can have been made via the monitor since it started. | 02:12 |
| SamYaple | Allen_Gao: yup | 02:15 |
| SamYaple | twas the case here | 02:15 |
| SamYaple | but now rabbitmq is breaking for me so i cant retest | 02:15 |
| *** inc0 has quit IRC | 02:41 | |
| *** tfukushima has quit IRC | 02:49 | |
| *** tfukushima has joined #kolla | 03:01 | |
| *** sdake has joined #kolla | 03:01 | |
| *** unicell has quit IRC | 03:06 | |
| *** yuanying has quit IRC | 03:15 | |
| *** yuanying has joined #kolla | 03:19 | |
| *** yuanying has quit IRC | 03:24 | |
| *** jtriley has quit IRC | 03:28 | |
| *** thumpba has joined #kolla | 03:34 | |
| *** yuanying has joined #kolla | 03:35 | |
| *** dims has joined #kolla | 03:37 | |
| *** yuanying_ has joined #kolla | 03:41 | |
| *** Jeffrey4l has joined #kolla | 03:42 | |
| *** farid has quit IRC | 03:43 | |
| *** yuanying has quit IRC | 03:44 | |
| *** sdake has quit IRC | 03:45 | |
| *** sdake has joined #kolla | 03:52 | |
| *** dims has quit IRC | 03:53 | |
| *** thumpba has quit IRC | 03:53 | |
| *** thumpba has joined #kolla | 03:54 | |
| *** salv-orl_ has joined #kolla | 03:54 | |
| *** salv-orlando has quit IRC | 03:58 | |
| *** yuanying_ has quit IRC | 04:06 | |
| *** yuanying has joined #kolla | 04:07 | |
| *** yuanying has quit IRC | 04:08 | |
| *** yuanying has joined #kolla | 04:08 | |
| *** jasonsb has quit IRC | 04:09 | |
| *** jasonsb has joined #kolla | 04:10 | |
| *** daneyon_ has joined #kolla | 04:11 | |
| *** macsz has quit IRC | 04:12 | |
| *** thumpba_ has joined #kolla | 04:12 | |
| *** daneyon has quit IRC | 04:12 | |
| *** tpeoples has quit IRC | 04:12 | |
| *** macsz has joined #kolla | 04:13 | |
| *** thumpba has quit IRC | 04:13 | |
| *** sdake has quit IRC | 04:13 | |
| *** Allen_Gao has quit IRC | 04:13 | |
| *** jpeeler has quit IRC | 04:13 | |
| *** emperor has quit IRC | 04:14 | |
| *** jasonsb has quit IRC | 04:15 | |
| *** jasonsb has joined #kolla | 04:16 | |
| *** tpeoples has joined #kolla | 04:16 | |
| *** bradjones has quit IRC | 04:19 | |
| *** elemoine has quit IRC | 04:19 | |
| *** Allen_Gao has joined #kolla | 04:19 | |
| *** Jeffrey4l_ has joined #kolla | 04:20 | |
| *** elemoine has joined #kolla | 04:20 | |
| *** jpeeler has joined #kolla | 04:21 | |
| *** bradjones has joined #kolla | 04:21 | |
| *** bradjones has quit IRC | 04:21 | |
| *** bradjones has joined #kolla | 04:21 | |
| *** thumpba_ has quit IRC | 04:25 | |
| *** tfukushima has quit IRC | 04:34 | |
| *** Allen_Gao has left #kolla | 05:04 | |
| *** Allen_Gao has joined #kolla | 05:04 | |
| *** dratushnyy has joined #kolla | 05:14 | |
| *** chandankumar has joined #kolla | 05:34 | |
| *** tfukushima has joined #kolla | 05:34 | |
| *** tfukushima has quit IRC | 05:40 | |
| *** sdake has joined #kolla | 05:40 | |
| dratushnyy | Hi | 05:43 |
| *** tfukushima has joined #kolla | 05:44 | |
| *** sdake_ has joined #kolla | 05:45 | |
| *** sdake has quit IRC | 05:45 | |
| kjelly | Hi dratushnyy | 05:49 |
| *** sdake_ has quit IRC | 06:13 | |
| *** sdake has joined #kolla | 06:13 | |
| *** dratushnyy has quit IRC | 06:17 | |
| *** dratushnyy has joined #kolla | 06:18 | |
| *** dratushnyy has quit IRC | 06:18 | |
| *** dave-mccowan has quit IRC | 06:30 | |
| *** dwalsh has joined #kolla | 06:33 | |
| *** Marga_ has joined #kolla | 06:56 | |
| *** dwalsh has quit IRC | 07:03 | |
| *** dratushnyy has joined #kolla | 07:12 | |
| *** salv-orl_ has quit IRC | 07:13 | |
| *** Jeffrey4l has quit IRC | 07:19 | |
| *** Jeffrey4l has joined #kolla | 07:32 | |
| *** unicell has joined #kolla | 07:37 | |
| *** sdake has quit IRC | 07:56 | |
| openstackgerrit | Merged openstack/kolla: Add double quotes in ceph_cache_mode https://review.openstack.org/275858 | 07:57 |
| *** fgimenez has joined #kolla | 08:06 | |
| *** Marga_ has quit IRC | 08:10 | |
| *** Marga_ has joined #kolla | 08:10 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 08:19 |
| *** dwalsh has joined #kolla | 08:21 | |
| *** akwasnie has joined #kolla | 08:23 | |
| *** gfidente has joined #kolla | 08:24 | |
| openstackgerrit | Merged openstack/kolla-mesos: Add requirements.txt to venv https://review.openstack.org/275916 | 08:25 |
| *** sdake has joined #kolla | 08:26 | |
| *** jmccarthy has joined #kolla | 08:27 | |
| *** neilus has quit IRC | 08:41 | |
| *** akwasnie has quit IRC | 08:44 | |
| *** neilus has joined #kolla | 08:44 | |
| *** ppetit has joined #kolla | 08:44 | |
| *** achanda has quit IRC | 08:45 | |
| *** akwasnie has joined #kolla | 08:46 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 08:48 |
| *** achanda has joined #kolla | 08:48 | |
| *** neilus has quit IRC | 08:49 | |
| *** Allen_Gao has quit IRC | 08:56 | |
| *** chandankumar has quit IRC | 08:56 | |
| *** rhallisey has joined #kolla | 09:04 | |
| *** dratushnyy has quit IRC | 09:09 | |
| *** Allen_Gao has joined #kolla | 09:12 | |
| *** macsz has quit IRC | 09:14 | |
| *** macsz has joined #kolla | 09:15 | |
| *** esikachev has joined #kolla | 09:15 | |
| ajafo | o/ | 09:15 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment. https://review.openstack.org/267714 | 09:17 |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #kolla | 09:17 | |
| akwasnie | ajafo: hi:) | 09:21 |
| *** mbound has joined #kolla | 09:24 | |
| elemoine | hi | 09:24 |
| akwasnie | hi elemoine | 09:26 |
| elemoine | hi akwasnie | 09:26 |
| elemoine | akwasnie, looking forwarding to seeing your ES code merged, to start Heka/ES/Kibana integration | 09:30 |
| *** dwalsh has quit IRC | 09:30 | |
| akwasnie | elemoine: me too and looking forward to see how heka works with ES and kibana!:) you've done a lot good work with heka already | 09:33 |
| elemoine | thanks | 09:33 |
| *** achanda has quit IRC | 09:33 | |
| elemoine | in the future we may need more specific configurations for ES and Kibana | 09:33 |
| elemoine | for example for ES we may want to define what fields are analyzed and not analyzed | 09:34 |
| akwasnie | yes, we should improve it, but I think that it can be done in next steps | 09:36 |
| elemoine | right, that's also the way I see it | 09:37 |
| akwasnie | I will be glad to help you with Heka/ES/Kibana integration and improvement, if you need any help | 09:38 |
| elemoine | excellent | 09:40 |
| elemoine | we can start the integration when ES is merged | 09:40 |
| akwasnie | yes | 09:41 |
| *** kjelly is now known as kejlly_ | 09:42 | |
| *** Jeffrey4l has quit IRC | 09:43 | |
| *** tzn has joined #kolla | 09:45 | |
| *** Jeffrey4l has joined #kolla | 10:01 | |
| *** gfidente has quit IRC | 10:10 | |
| *** gfidente has joined #kolla | 10:15 | |
| *** gfidente has quit IRC | 10:15 | |
| *** gfidente has joined #kolla | 10:15 | |
| *** dwalsh has joined #kolla | 10:27 | |
| *** Jeffrey4l has quit IRC | 10:29 | |
| *** neilus has joined #kolla | 10:29 | |
| *** tfukushima has quit IRC | 10:30 | |
| *** tfukushima has joined #kolla | 10:32 | |
| *** neilus has quit IRC | 10:35 | |
| *** dwalsh has quit IRC | 10:40 | |
| *** dwalsh has joined #kolla | 10:40 | |
| *** fgimenez has quit IRC | 10:41 | |
| *** fgimenez has joined #kolla | 10:42 | |
| *** sdake has quit IRC | 10:44 | |
| *** tzn has quit IRC | 10:48 | |
| *** Allen_Gao has quit IRC | 10:48 | |
| *** esikachev has quit IRC | 10:52 | |
| *** esikachev has joined #kolla | 10:53 | |
| *** akwasnie has quit IRC | 10:53 | |
| *** salv-orlando has joined #kolla | 10:54 | |
| *** dwalsh has quit IRC | 11:01 | |
| *** tfukushima has quit IRC | 11:04 | |
| *** Allen_Gao has joined #kolla | 11:06 | |
| *** dims has joined #kolla | 11:12 | |
| *** egonzalez90 has joined #kolla | 11:23 | |
| *** egonzalez90 is now known as egonzalez | 11:24 | |
| openstackgerrit | Merged openstack/kolla: gnocchi is busted missing tooz for source https://review.openstack.org/275898 | 11:24 |
| *** ppetit has quit IRC | 11:25 | |
| *** kproskurin has joined #kolla | 11:25 | |
| openstackgerrit | Merged openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 11:31 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 11:36 |
| pbourke | SamYaple: can we sway you on https://review.openstack.org/#/c/268211 | 11:37 |
| openstackgerrit | MD NADEEM proposed openstack/kolla: Update kernel version in ubuntu-bootstrap.sh https://review.openstack.org/276178 | 11:42 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "fix miss rabbitmq config when hostname include special character" https://review.openstack.org/276180 | 11:47 |
| SamYaple | uughhhhhh | 11:47 |
| SamYaple | pbourke: Jeffrey4l_ you both approved the initial patch, can you approve the revert? https://review.openstack.org/#/c/276180/ | 11:48 |
| SamYaple | i never even saw the patch in the queue or I would hae blocked :/ | 11:48 |
| SamYaple | this is why my rabbitmq has been busted for two days | 11:48 |
| pbourke | :( | 11:48 |
| pbourke | who cares about ubuntu | 11:48 |
| SamYaple | ohly everyone | 11:48 |
| pbourke | why did all the gates go green | 11:49 |
| SamYaple | it breaks on certain ubuntu kernels | 11:50 |
| SamYaple | because erlang has direct kernel talkings | 11:50 |
| *** fgimenez has quit IRC | 11:51 | |
| SamYaple | pbourke: https://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.config.j2#L2 | 11:52 |
| *** fgimenez has joined #kolla | 11:53 | |
| *** fgimenez has quit IRC | 11:53 | |
| *** fgimenez has joined #kolla | 11:53 | |
| SamYaple | pbourke: official guide has no quotes, no quotes works for me, quotes breaks for me. need the revert | 11:54 |
| pbourke | yup that's fine | 11:55 |
| pbourke | unfortunate for the guy who has quotes in his hostname though | 11:55 |
| SamYaple | i have dashes in my hostname... | 11:56 |
| SamYaple | https://bugs.launchpad.net/kolla/+bug/1540234 | 11:56 |
| openstack | Launchpad bug 1540234 in kolla "rabbitmq cluster is not work" [Undecided,Confirmed] - Assigned to weiyu (weiyu) | 11:56 |
| SamYaple | see last comment | 11:56 |
| SamYaple | its literally the inverse of the above comments | 11:56 |
| pbourke | ok that's good to know | 11:56 |
| SamYaple | he is using centos though | 11:57 |
| limamauricio | Can I deploy ceph all-in-one? | 11:58 |
| limamauricio | All running only in one vm | 11:59 |
| SamYaple | limamauricio: yes, but it requires a bit more knowleged about ceph | 11:59 |
| limamauricio | :/ | 11:59 |
| SamYaple | specfically you need to set the replicas to 1 or update teh failure domain of the object to osd instead of host | 11:59 |
| limamauricio | Sam, can you see this error http://pastebin.com/zgvCppsy ? | 12:02 |
| limamauricio | I'm following this guide http://docs.openstack.org/developer/kolla/ceph-guide.html | 12:02 |
| limamauricio | but I have it now :/ | 12:02 |
| SamYaple | limamauricio: are you on a completely clean environment? no leftover contaienrs, configs, or volumes from a previous deploy? | 12:02 |
| SamYaple | `docker volume ls` | 12:03 |
| SamYaple | `ls /etc/kolla` | 12:03 |
| SamYaple | `docker ps -a` | 12:03 |
| *** neilus has joined #kolla | 12:03 | |
| SamYaple | once you get the evironment clean the above commands shouldn't show _any_ reference to ceph | 12:03 |
| limamauricio | I' have to clean all? | 12:03 |
| SamYaple | at least teh ceph stuff | 12:03 |
| SamYaple | what you see if from a failed parital deploy | 12:04 |
| limamauricio | Ok | 12:04 |
| *** athomas has joined #kolla | 12:07 | |
| *** neilus has quit IRC | 12:08 | |
| *** Allen_Gao has quit IRC | 12:17 | |
| *** dwalsh has joined #kolla | 12:19 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Make libvirt reestablish connection with qemu https://review.openstack.org/276193 | 12:22 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 12:27 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Switch from mysql-python to pymysql everywhere https://review.openstack.org/276194 | 12:27 |
| *** Allen_Gao has joined #kolla | 12:30 | |
| limamauricio | Sam, monitor data directory at '/var/lib/ceph/mon/ceph-localhost' does not exist: have you run 'mkfs'? | 12:32 |
| *** salv-orlando has quit IRC | 12:32 | |
| limamauricio | i ran mkfs.ext4 | 12:32 |
| limamauricio | mkfs.ext4 /dev/sdb :/ | 12:32 |
| *** iceyao has joined #kolla | 12:34 | |
| SamYaple | limamauricio: no thats for the osd | 12:34 |
| SamYaple | the mon doesnt need a disk like that | 12:35 |
| limamauricio | I need set enable_ceph_rgw: "yes"? | 12:36 |
| limamauricio | and create /etc/kolla/config/ceph.conf file? | 12:37 |
| SamYaple | if you want to use the radosgw you do | 12:37 |
| SamYaple | but its not required | 12:37 |
| limamauricio | ok | 12:38 |
| *** iceyao has quit IRC | 12:38 | |
| openstackgerrit | MD NADEEM proposed openstack/kolla: Update kernel version in ubuntu-bootstrap.sh https://review.openstack.org/276178 | 12:41 |
| *** salv-orlando has joined #kolla | 12:46 | |
| *** mkoderer has quit IRC | 12:49 | |
| *** vincent_1dk is now known as vincent_vdk | 12:49 | |
| *** vincent_vdk has quit IRC | 12:49 | |
| *** vincent_vdk has joined #kolla | 12:49 | |
| *** elemoine- has joined #kolla | 12:52 | |
| *** dave-mccowan has joined #kolla | 12:53 | |
| *** mdnadeem has quit IRC | 12:53 | |
| *** elemoine has quit IRC | 12:53 | |
| *** mdnadeem has joined #kolla | 12:54 | |
| *** egonzalez has quit IRC | 12:58 | |
| *** mkoderer has joined #kolla | 13:03 | |
| *** iceyao has joined #kolla | 13:04 | |
| *** dwalsh has quit IRC | 13:08 | |
| *** iceyao has quit IRC | 13:14 | |
| *** inc0 has joined #kolla | 13:21 | |
| inc0 | hey good morning | 13:21 |
| inc0 | SamYaple, up and about already? | 13:22 |
| *** dwalsh has joined #kolla | 13:25 | |
| limamauricio | Someone did kolla-ansible deploy with ceph works? | 13:28 |
| limamauricio | all-in-one | 13:29 |
| inc0 | aio ceph needs additional config | 13:29 |
| limamauricio | I'm following this guide http://docs.openstack.org/developer/kolla/ceph-guide.html | 13:29 |
| limamauricio | I never worked with ceph | 13:30 |
| inc0 | at the bottom there is paragraph about aio | 13:30 |
| inc0 | in Managing ceph | 13:31 |
| limamauricio | When I ran kolla-ansible deploy I always get the same error | 13:31 |
| limamauricio | TASK: [glance | Pulling cephx keyring] | 13:31 |
| inc0 | because ceph doesn't work in aio without additional config, I just said that:P | 13:31 |
| inc0 | that's normal and expected behaviour | 13:31 |
| inc0 | ceph normally isn't prepared to be aio | 13:31 |
| inc0 | you can do this by following ceph guide, there is paragraph in "Managing ceph" about that | 13:32 |
| limamauricio | ok | 13:32 |
| *** gfidente has quit IRC | 13:39 | |
| openstackgerrit | venkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276230 | 13:43 |
| *** gfidente has joined #kolla | 13:47 | |
| *** gfidente has quit IRC | 13:47 | |
| *** gfidente has joined #kolla | 13:47 | |
| *** Jeffrey4l has joined #kolla | 13:48 | |
| *** sdake has joined #kolla | 13:50 | |
| *** neilus has joined #kolla | 13:51 | |
| *** neilus has quit IRC | 13:56 | |
| *** Allen_Gao has quit IRC | 13:57 | |
| mdnadeem | ajafo, ping | 13:58 |
| mdnadeem | ajafo, I am new to kolla and looking for to start contribution in kolla, would you mind if i can help you to complete this bp https://blueprints.launchpad.net/kolla/+spec/binary-ubuntu ? | 14:01 |
| limamauricio | Did someone here deploy ceph aio successfully? If yes, please tell me how. | 14:03 |
| *** dwalsh has quit IRC | 14:03 | |
| limamauricio | I did the additional config, but not works. | 14:04 |
| *** barra204 has quit IRC | 14:04 | |
| *** dwalsh has joined #kolla | 14:06 | |
| *** shakamunyi has joined #kolla | 14:06 | |
| *** fgimenez has quit IRC | 14:06 | |
| *** fgimenez has joined #kolla | 14:10 | |
| *** fgimenez has quit IRC | 14:10 | |
| *** fgimenez has joined #kolla | 14:10 | |
| *** dims_ has joined #kolla | 14:10 | |
| ajafo | mdnadeem: I had to leave this bp, at the moment but sure if you can it would be nice | 14:11 |
| ajafo | mdnadeem: this can be little helpful https://review.openstack.org/#/c/260069/ | 14:11 |
| *** dims has quit IRC | 14:12 | |
| inc0 | SamYaple, so quick question, I did spawn vm, killed docker and started docker, vm dies as any other process | 14:17 |
| inc0 | if it is not the case for you, maybe they 'broke' something in 1.10 which helps | 14:18 |
| *** tzn has joined #kolla | 14:20 | |
| *** salv-orlando has quit IRC | 14:20 | |
| rhallisey | inc0, does this only happen in 1.10? | 14:21 |
| inc0 | not sure | 14:22 |
| inc0 | I'm on 1.9 and vm dies | 14:22 |
| rhallisey | inc0, reason I'm asking is I probably going to help the RHT docker people debug this and I'd like to pin the version | 14:22 |
| inc0 | Sam told me yesteday that it does not | 14:22 |
| inc0 | there might be other causes tho, not sure yet | 14:22 |
| rhallisey | inc0, kk | 14:23 |
| inc0 | we need to pinpoint it | 14:23 |
| inc0 | about version pin, you will need at least 1.10 | 14:23 |
| inc0 | docker | 14:23 |
| inc0 | is it available in centos? | 14:23 |
| inc0 | I mean will it be by kolla release? | 14:23 |
| *** salv-orlando has joined #kolla | 14:24 | |
| rhallisey | sorry didn't mean pin the version like that | 14:24 |
| *** tzn has quit IRC | 14:24 | |
| inc0 | thin neutron container forces at least 1.10 | 14:24 |
| rhallisey | pinpoint would be more accurate | 14:24 |
| inc0 | ok | 14:24 |
| rhallisey | but pin agree if there is an issue | 14:24 |
| inc0 | yeah, we need to check that one out | 14:24 |
| openstackgerrit | venkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276230 | 14:24 |
| *** tzn has joined #kolla | 14:26 | |
| sdake | SamYaple around? | 14:27 |
| sdake | inc0 SamYaple can both of you look at https://bugs.launchpad.net/kolla/+bug/1541862 | 14:27 |
| openstack | Launchpad bug 1541862 in kolla "stable/liberty vagrant aio fails at "Creating haproxy mysql user"" [Undecided,New] | 14:27 |
| inc0 | Chris already answered and it seems to be good, let me double-check tho | 14:28 |
| inc0 | ah | 14:29 |
| inc0 | wrong, it seems he pinpointed problem but vagrant need fixing | 14:29 |
| *** tzn has quit IRC | 14:35 | |
| sdake | i see its a dupe | 14:37 |
| sdake | i dont think vagrant has anything to do with it | 14:37 |
| sdake | here is the master dupe https://bugs.launchpad.net/kolla/+bug/1541862 | 14:37 |
| openstack | Launchpad bug 1541862 in kolla "stable/liberty fails at "Creating haproxy mysql user" because of module error" [Critical,Triaged] | 14:37 |
| sdake | we need a stable target in launchpad i guess | 14:38 |
| *** ftcpops has joined #kolla | 14:49 | |
| inc0 | I should start using vagrant | 15:00 |
| inc0 | I keep hearing lots of problems with it and can't help with them | 15:00 |
| sdake | half our dev team uses vagrant | 15:01 |
| sdake | but inc0 i feel the same way | 15:01 |
| *** daneyon_ has quit IRC | 15:01 | |
| inc0 | I never did and old habits die hard I guess | 15:01 |
| sdake | inc0 https://bugs.launchpad.net/kolla/liberty/+bug/1530962 | 15:04 |
| openstack | Launchpad bug 1530962 in kolla liberty "mariadb "Creating haproxy mysql user" task failing in stable/liberty" [Critical,Confirmed] | 15:04 |
| pbourke | inc0: there's a reason you keep hearing lots of problems with it ;) | 15:05 |
| sdake | https://bugs.launchpad.net/kolla/liberty/+bug/1530962 | 15:05 |
| dmsimard | nihilifer: that's your commit, right ? https://review.openstack.org/#/c/233939/ (sorry if it's another Michal) | 15:06 |
| inc0 | yeah its nihilifer | 15:06 |
| dmsimard | there are still build failures in gate jobs that this commit was supposed to prevent | 15:08 |
| dmsimard | and funny thing is that jenkins is still returning +1 | 15:08 |
| dmsimard | here's an example: http://logs.openstack.org/63/275463/3/check/gate-kolla-dsvm-build-centos-binary/1a707b4/console.html#_2016-02-03_16_44_39_096 (scroll to the end) | 15:08 |
| inc0 | sdake, https://github.com/openstack/kolla/blob/stable/liberty/requirements.txt we don't even have six there | 15:08 |
| dmsimard | am I missing something here ? | 15:08 |
| dmsimard | I just looked at the centos binary job from https://review.openstack.org/#/c/275463/ | 15:09 |
| dmsimard | http://logs.openstack.org/63/275463/3/check/gate-kolla-dsvm-build-centos-binary/1a707b4/console.html#_2016-02-03_16_27_45_997 is returning failure because mistral is not available as centos binary so I would've expected https://github.com/openstack/kolla/blob/75343870e395e257c8d8267ccd9c74e70c7fceb5/tests/test_build.py#L74 to prevent that from building in the first place | 15:11 |
| dmsimard | or wait, it doesn't prevent it from building, it ignores if there is an error building it | 15:12 |
| sdake | inc0 someone backported something requiring six | 15:13 |
| sdake | that is option a | 15:13 |
| sdake | or option b is | 15:13 |
| sdake | the kolla-ansible dockerfile six deepndency changed in some way | 15:14 |
| sdake | SamYaple used a specific tag for ansible | 15:14 |
| sdake | so it should never break | 15:14 |
| sdake | but it is | 15:14 |
| inc0 | one of deps in requirements might have dependency on six and version changed | 15:15 |
| inc0 | bottom line, we need to find version of six that works and pin to it | 15:15 |
| sdake | inc0 its happening in the kolla_ansible module | 15:18 |
| sdake | not in ansible | 15:18 |
| *** asalkeld has quit IRC | 15:20 | |
| inc0 | I'd say shade is problematic | 15:20 |
| inc0 | we dont explicitly install six in kolla-ansible, but we use shade and we don't pin version on it | 15:20 |
| *** mimizone has joined #kolla | 15:21 | |
| sdake | there is a worka round in the bug | 15:21 |
| openstackgerrit | Evgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py https://review.openstack.org/276298 | 15:24 |
| *** blahRus has joined #kolla | 15:26 | |
| *** fgimenez has quit IRC | 15:28 | |
| *** akwasnie has joined #kolla | 15:28 | |
| *** fgimenez has joined #kolla | 15:28 | |
| *** achanda has joined #kolla | 15:30 | |
| *** jtriley has joined #kolla | 15:37 | |
| inc0 | sdake, I see it now | 15:38 |
| limamauricio | SamYaple, now i have an error in this task: [ceph | Bootstrapping Ceph OSDs] | 15:38 |
| inc0 | we install git ansible instead of pip install ansible | 15:38 |
| limamauricio | :/ | 15:38 |
| inc0 | and it was alpha | 15:38 |
| limamauricio | i'm using centos | 15:39 |
| inc0 | ansible 2.0 is released so now we should just install it with pip | 15:39 |
| *** neilus has joined #kolla | 15:40 | |
| *** achanda has quit IRC | 15:42 | |
| limamauricio | failed: [localhost] => (item=(0, {u'device': u'/dev/sdb', u'fs_uuid': u''})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "fs_uuid": ""}]} msg: Container exited with non-zero return code | 15:42 |
| *** jtriley has quit IRC | 15:43 | |
| *** dratushnyy has joined #kolla | 15:44 | |
| *** neilus has quit IRC | 15:44 | |
| *** tzn has joined #kolla | 15:46 | |
| *** alyson_ has joined #kolla | 15:48 | |
| *** alyson_ has quit IRC | 15:48 | |
| *** JoseMello has joined #kolla | 15:48 | |
| *** alyson_ has joined #kolla | 15:49 | |
| *** tzn has quit IRC | 15:57 | |
| SamYaple | inc0: ping | 15:58 |
| inc0 | SamYaple, here, qemu gets restarted for me :/ | 15:58 |
| *** jtriley has joined #kolla | 15:58 | |
| SamYaple | inc0: I have 100% confirmed on latest head it does _not_ get restarted at all for me | 15:59 |
| SamYaple | qemu processes stick around | 15:59 |
| SamYaple | world is good | 15:59 |
| inc0 | I believe you, I wonder what's wrong with my case | 15:59 |
| inc0 | maybe they changes something in 1.10 | 15:59 |
| inc0 | I'm running 1.9 | 15:59 |
| SamYaple | no i checked 1.9 as well | 15:59 |
| SamYaple | and 1.8 for that matter | 16:00 |
| inc0 | so here is what I do, I run vm, ps aux | grep qemu | 16:00 |
| inc0 | then I watch vm pid | 16:00 |
| inc0 | if I do docker rm -f nova_libvirt pid dies | 16:00 |
| inc0 | if I kill -9 docker, pid stays, but then if I start docker, pid dies | 16:00 |
| SamYaple | does pid actually die, or does it get a new pid? | 16:00 |
| SamYaple | does qemu process survive is my question | 16:00 |
| inc0 | I'm checking pid | 16:01 |
| *** esikachev has quit IRC | 16:01 | |
| inc0 | also libvirt shows qemu instance as down | 16:01 |
| inc0 | after restart of container | 16:01 |
| SamYaple | i have a patch for that | 16:01 |
| inc0 | I've seen, +2 it | 16:01 |
| inc0 | I was sure we do share /run | 16:01 |
| inc0 | well we used to;) | 16:02 |
| inc0 | SamYaple, you use qemu+kvm or just qemu? | 16:02 |
| SamYaple | inc0: +kvm | 16:03 |
| SamYaple | and as i thought ll along _that_ might be the key | 16:03 |
| SamYaple | the kernel tie in | 16:04 |
| SamYaple | but if that is the case maybe we can replicate it | 16:04 |
| inc0 | I run just qemu, so yeah | 16:04 |
| SamYaple | i can test just kvm | 16:04 |
| SamYaple | just qemu* | 16:04 |
| inc0 | yeah, do that | 16:04 |
| inc0 | trying just kvm would be hard;) | 16:04 |
| SamYaple | hey can you give me a +2 on this revert inc0 ? Its killing me and my testing right now https://review.openstack.org/#/c/276180/ | 16:05 |
| sdake | 2142 commits in liberty, 2896 commits in master | 16:05 |
| sdake | our velocity is off the hook | 16:05 |
| inc0 | we have merge fest tho:P | 16:05 |
| sdake | inc0 | 16:06 |
| sdake | shall I take careof that ansible 2.0.0 thing then? | 16:06 |
| sdake | I need it done asap | 16:06 |
| inc0 | I'm rebuilding containers now | 16:06 |
| sdake | or are you working on it already | 16:06 |
| inc0 | for tests | 16:06 |
| SamYaple | sdake: what ansible thing? | 16:06 |
| inc0 | SamYaple, https://bugs.launchpad.net/kolla/liberty/+bug/1530962 | 16:06 |
| openstack | Launchpad bug 1530962 in kolla liberty "six version dependency in kolla_ansible appears to be incorrect." [Critical,Confirmed] | 16:06 |
| sdake | SamYaple https://bugs.launchpad.net/kolla/liberty/+bug/1530962 | 16:06 |
| sdake | yup | 16:07 |
| SamYaple | oh yea ive seen that | 16:07 |
| sdake | stable is broken | 16:07 |
| inc0 | I'm pinning ansible to 2.0.0.2 and will test master | 16:07 |
| sdake | have demo of stable in 3 days | 16:07 |
| SamYaple | thats from an ordering issue | 16:07 |
| SamYaple | on installs | 16:07 |
| SamYaple | see https://review.openstack.org/#/c/268863/ | 16:07 |
| inc0 | anyway, do we still need to keep ansible from source? | 16:08 |
| SamYaple | yea | 16:08 |
| SamYaple | ansible 2.0 is broken with mysql | 16:09 |
| SamYaple | more specifically | 16:09 |
| SamYaple | its broken with the way we do the grants | 16:09 |
| sdake | thanks SamYaple | 16:09 |
| SamYaple | they broke it in rc8 (again) | 16:09 |
| *** kproskurin has quit IRC | 16:09 | |
| SamYaple | ive already fixed the issue once | 16:09 |
| SamYaple | https://github.com/ansible/ansible-modules-core/pull/1934 | 16:09 |
| inc0 | mandre, already in France?:) | 16:10 |
| sdake | inc0 here is the problem https://review.openstack.org/#/c/268863/2 | 16:13 |
| sdake | it would be interesting to understand what the real issue is - if only there were a way to record information in the bug tracker about what was happening with a bug ;) | 16:14 |
| SamYaple | sdake: mandre describes it | 16:14 |
| sdake | changing orders of install isn't going to do a damn thing | 16:14 |
| inc0 | sdake, mandre said that it does not fix issue | 16:14 |
| inc0 | in review | 16:14 |
| SamYaple | six has always been a dep | 16:14 |
| *** farid_ has joined #kolla | 16:14 | |
| SamYaple | its just conflicting | 16:14 |
| SamYaple | inc0: but its an ordering thing still | 16:15 |
| SamYaple | it fixes it when installed manually | 16:15 |
| sdake | it does but not automatically? | 16:15 |
| SamYaple | sdake: it _will_ fix it chaning order of installs | 16:15 |
| sdake | oh you mean not with pip -r requirements.txt | 16:15 |
| SamYaple | thats like 90% of teh pip issue | 16:15 |
| SamYaple | not just that | 16:15 |
| SamYaple | but with host conflicts as well | 16:16 |
| sdake | my understanding this is not on the host but this is on the container | 16:16 |
| sdake | is that correct? | 16:16 |
| SamYaple | this is a dep problem in the container, yes | 16:17 |
| sdake | reading pypi bug report | 16:17 |
| SamYaple | honestly i think we can just bump ansible to master version.... let me check | 16:18 |
| sdake | will that fix this problem? | 16:18 |
| sdake | you mean 2.0.0 version | 16:18 |
| SamYaple | no | 16:19 |
| inc0 | 2.0.0.2 is current pypi latest | 16:19 |
| SamYaple | we cannot bump | 16:19 |
| SamYaple | please read my comments above | 16:19 |
| inc0 | I know | 16:19 |
| SamYaple | i meant what we have in master | 16:19 |
| inc0 | ah so backport ansible version from our current master? | 16:20 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Bump liberty kolla-ansible continer ansible https://review.openstack.org/276328 | 16:20 |
| *** dwalsh has quit IRC | 16:21 | |
| SamYaple | sdake: inc0 ^ | 16:21 |
| SamYaple | thats a safe operation for liberty since no changes occured | 16:21 |
| SamYaple | anything rc8 and up will break our playbooks | 16:21 |
| inc0 | and you think it will fix issue at hand? | 16:21 |
| SamYaple | since we don't see it in master, i would say yes | 16:23 |
| *** dwalsh has joined #kolla | 16:23 | |
| sdake | SamYaple let me test it | 16:23 |
| SamYaple | i have a patch for bumping kolla-ansible's ansible to non-rc version, but like i said it breaks our playbooks | 16:23 |
| sdake | if it tests out, i'll approve | 16:23 |
| SamYaple | i would have to adjust it all | 16:23 |
| sdake | let me test the review in queue not the ansible 2.0.0 thing | 16:24 |
| SamYaple | inc0: was that a no on the revert merge? It kills me since it breaks my deploys and i have to manually fix when testing this qemu stuff https://review.openstack.org/#/c/276180/ | 16:24 |
| inc0 | SamYaple, I just want to test it | 16:24 |
| inc0 | I try to figure out what happend | 16:25 |
| SamYaple | \' is a special char in rabbitmq config | 16:25 |
| inc0 | did someone break our configs and we did merge it? | 16:25 |
| SamYaple | yup | 16:25 |
| inc0 | dug | 16:25 |
| SamYaple | look at the commit im reverting | 16:25 |
| inc0 | duh | 16:25 |
| limamauricio | SamYaple, now i have an error in this task: [ceph | Bootstrapping Ceph OSDs] | 16:25 |
| limamauricio | failed: [localhost] => (item=(0, {u'device': u'/dev/sdb', u'fs_uuid': u''})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "fs_uuid": ""}]} msg: Container exited with non-zero return code | 16:25 |
| SamYaple | limamauricio: you should have a boostrap_osd_* container | 16:26 |
| SamYaple | look at the logs from that container | 16:26 |
| SamYaple | `docker logs bootstrap_osd_*` | 16:26 |
| inc0 | SamYaple, what happend to our CI? | 16:27 |
| inc0 | no functional tests after deploy? | 16:27 |
| sdake | inc0 we are working on it | 16:27 |
| inc0 | also I did deploy succesfully on ubuntu today | 16:27 |
| inc0 | with vm booting | 16:28 |
| inc0 | and all | 16:28 |
| sdake | inc0 the problem is ram, I've got a couple cats sorting out the ram issue as best they can | 16:28 |
| SamYaple | inc0: this only affects hostnames with certain chars | 16:28 |
| SamYaple | sdake: thats not related | 16:28 |
| inc0 | ok.. | 16:28 |
| SamYaple | inc0: look at bug, its all there | 16:28 |
| inc0 | what chars are these?:P | 16:28 |
| sdake | if i seem incoherent, 3 hrs of sleep for 2 weeks + 12 hours of manual labor every day will do that to you :) | 16:29 |
| inc0 | nah that was me being unclear | 16:29 |
| sdake | i am so exhausted | 16:30 |
| sdake | and in so much pain i just want to cry | 16:30 |
| *** shardy has joined #kolla | 16:30 | |
| SamYaple | sdake: you dont have to be here | 16:30 |
| sdake | i have definately put my body throug hthe ringer | 16:30 |
| SamYaple | you can sleep | 16:30 |
| SamYaple | youll get more done if you sleep first | 16:30 |
| sdake | i did get a night of sleep last night full 10 hours | 16:30 |
| sdake | it was glorious | 16:30 |
| sdake | i honestly can't sleep anymore today | 16:30 |
| sdake | ther eason iwas getting 3 hrs of sleep is the contractor dudes show up at 4am-5am | 16:31 |
| inc0 | sdake, man you are breaking apart this release;) | 16:31 |
| sdake | and I go to bed at midnight gettting prepped for the next day | 16:31 |
| sdake | inc0 how do you mean | 16:31 |
| inc0 | first tooth, now sleep depravation | 16:31 |
| sdake | the good thing is, once this remodel is done (2 weeks) I'm going to be back at full 18 hr days of work I actually like doing - yay :) | 16:32 |
| *** esikachev has joined #kolla | 16:32 | |
| inc0 | get some sleep first;) | 16:32 |
| sdake | not tired | 16:32 |
| sdake | body tired | 16:32 |
| sdake | i have full r&r before midcycle | 16:32 |
| sdake | so should be at 100% for that :) | 16:32 |
| sdake | I'll need it to keep up with you youngins :) | 16:33 |
| inc0 | yeah I'm going to SC Saturday and will stay over a week there | 16:33 |
| inc0 | for r&r | 16:33 |
| sdake | i am in monday-wed | 16:33 |
| sdake | i leave wed at 3:30 pm | 16:33 |
| openstackgerrit | Merged openstack/kolla: Revert "fix miss rabbitmq config when hostname include special character" https://review.openstack.org/276180 | 16:33 |
| inc0 | SamYaple, try out qemu plz | 16:34 |
| sdake | thats so the folks on the west coast can get back home without getting stuck for another hotel night | 16:34 |
| SamYaple | inc0: wow 14 seconds after merge you ask | 16:34 |
| SamYaple | cant even git pull htat fast | 16:34 |
| inc0 | give some amphetamine to your packet-carrying-pidgeons | 16:35 |
| sdake | i had a friend that used crystal meth when he started working | 16:35 |
| sdake | he blew up a lawyers car and went to jail | 16:35 |
| sdake | he himself was an attorney | 16:35 |
| sdake | got disbarred | 16:35 |
| sdake | his life was a disaster for about 15 years | 16:36 |
| sdake | I stil ldont think hes clean | 16:36 |
| sdake | amphetemine is not to be touched :) | 16:36 |
| inc0 | yeah, for pidgeons tho... | 16:36 |
| pbourke | sdake: "friend" | 16:36 |
| inc0 | lsd is better for programming;) | 16:36 |
| pbourke | ;) | 16:37 |
| sdake | pbourke best friend in fact | 16:37 |
| *** salv-orlando has quit IRC | 16:37 | |
| sdake | i really tried to support him for a long time | 16:37 |
| sdake | but eventually gave up on him | 16:37 |
| sdake | i dont think lsd and writing code woudl work out | 16:37 |
| sdake | but whatever your poison :) | 16:38 |
| openstackgerrit | Merged openstack/kolla: Make libvirt reestablish connection with qemu https://review.openstack.org/276193 | 16:38 |
| sdake | Mine is Nuvigil | 16:38 |
| inc0 | lsd and architectural problem solving does | 16:38 |
| sdake | https://en.wikipedia.org/wiki/Armodafinil | 16:38 |
| sdake | i tried mushrooms on college | 16:38 |
| sdake | never lsd | 16:39 |
| sdake | i think they are similiar | 16:39 |
| inc0 | they are not | 16:39 |
| sdake | i definately coudln't do anything on mushrooms | 16:39 |
| inc0 | mushrooms are halucinogen, lsd is psychodelic | 16:39 |
| sdake | mushrooms rocked but i wouldn't do that again | 16:39 |
| sdake | dude they are both hallucingens :) | 16:39 |
| *** esikachev has quit IRC | 16:40 | |
| inc0 | not really, lsd changes perception but don't add new stuff to it...at least for me | 16:40 |
| limamauricio | SamYaple, log from logs bootstrap_osd_0 http://pastebin.com/n8XKvd7q | 16:40 |
| inc0 | but I did hear Mozart in 3d ;) | 16:40 |
| *** ppetit has joined #kolla | 16:40 | |
| inc0 | and I mean spatial 3d not just stereo | 16:41 |
| SamYaple | limamauricio: looking | 16:41 |
| SamYaple | limamauricio: it cannot connect to your ceph-mon | 16:41 |
| SamYaple | limamauricio: is this an AIO? | 16:41 |
| limamauricio | yes | 16:41 |
| limamauricio | aio | 16:41 |
| SamYaple | is the ceph-mon running? | 16:42 |
| SamYaple | this is master branch too correct?) | 16:42 |
| limamauricio | wait | 16:42 |
| limamauricio | 82b41e32a35f localhost:4000/kollaglue/centos-source-ceph-mon:2.0.0 "kolla_start" 8 minutes ago Restarting (1) About a minute ago ceph_mon | 16:42 |
| limamauricio | no :/ | 16:42 |
| SamYaple | youll want to check that log | 16:42 |
| SamYaple | see whats up | 16:42 |
| limamauricio | ok | 16:42 |
| limamauricio | Sam http://pastebin.com/JyNfJW5B | 16:43 |
| inc0 | what version of docker you're running? | 16:44 |
| inc0 | limamauricio, | 16:44 |
| limamauricio | Docker version 1.9.1, build a34a1d5 | 16:44 |
| limamauricio | I'm trying multinode too | 16:45 |
| *** fgimenez has quit IRC | 16:45 | |
| SamYaple | limamauricio: you need to pick one patch and stick with it for now | 16:45 |
| SamYaple | this is all caused by having a dirty environtment | 16:45 |
| SamYaple | i would recommend a full clean of containers, configs, and docker volumes | 16:45 |
| inc0 | and remember to re-partition your osd disk | 16:45 |
| *** fgimenez has joined #kolla | 16:47 | |
| limamauricio | okay :( | 16:47 |
| limamauricio | I will do everything from scratch | 16:47 |
| SamYaple | limamauricio: you dont need to rebuild your container images, but you can | 16:47 |
| pbourke | SamYaple: did we not backport kolla_ansible to liberty | 16:48 |
| inc0 | do it just in case | 16:48 |
| SamYaple | you do need to clean everythign else, all ceph configs in /etc/kolla, all containers for ceph deploy, all ceph volumes in `docker volume ls` | 16:48 |
| pbourke | I mean kolla_docker | 16:48 |
| SamYaple | pbourke: nooo | 16:48 |
| limamauricio | I cleaned everything and i have the same error | 16:48 |
| inc0 | and we shouldn't pbourke | 16:48 |
| SamYaple | thats a very large change in playbooks across the board | 16:48 |
| inc0 | too big of a change | 16:48 |
| SamYaple | its like a 30% change | 16:49 |
| pbourke | inc0: upgrade testing is a bi**h | 16:49 |
| inc0 | that's what people will experience | 16:49 |
| inc0 | so yeah I do know that | 16:49 |
| pbourke | inc0: as you have to swap versions of docker/docker-py | 16:49 |
| inc0 | yup | 16:49 |
| pbourke | hmm | 16:49 |
| inc0 | and try to do this without tearing down your vms | 16:49 |
| pbourke | >_< | 16:49 |
| inc0 | that's why I'm freaking out whole week | 16:50 |
| SamYaple | i can tear it down with out hitting vms :P | 16:50 |
| inc0 | SamYaple, we're getting there | 16:50 |
| inc0 | if it works for qemu+kvm I'm ok with that | 16:50 |
| *** rhallisey has quit IRC | 16:50 | |
| inc0 | if someone is running qemu he shouldn't care about vm downtime anyway | 16:50 |
| *** sdake has quit IRC | 16:51 | |
| *** sdake has joined #kolla | 16:52 | |
| SamYaple | wow youre just generalizing! | 16:52 |
| SamYaple | nice | 16:52 |
| *** shardy has quit IRC | 16:52 | |
| inc0 | well, have you ever seen qemu without kvm prod? | 16:52 |
| inc0 | and performance of these vms would be so low that I'm not even afraid about LM-ing them, even over pidgeons on crystal meth | 16:53 |
| *** dwalsh has quit IRC | 16:56 | |
| sdake | SamYaple did you confirm that restart of containers keeps qemu-kvm active? | 16:57 |
| sdake | because a couple days ago you said that was not the case in your testing | 16:57 |
| SamYaple | inc0: please run `ps -p 2666 -o f` where 2666 is a qemu without kvm process running from container | 16:57 |
| SamYaple | sdake: i said processes i did not say qemu specifically (didnt test qemu) | 16:58 |
| sdake | ok noted | 16:58 |
| sdake | so very likely qemu is still operational | 16:58 |
| sdake | oh | 16:58 |
| sdake | duh | 16:58 |
| sdake | got it you tried different processes | 16:58 |
| sdake | SamYaple what processes did you see were retained on a libvirt restart? | 16:59 |
| inc0 | sdake, this is required for libvirt to reconnect to instances: https://review.openstack.org/#/c/276193/ | 17:02 |
| SamYaple | inc0: that merged, but was a master specfic patch | 17:02 |
| SamYaple | inc0: what was the output of the command? | 17:02 |
| sdake | i knew that change was wrong inc0 | 17:02 |
| sdake | i even left a comment about it :) | 17:03 |
| inc0 | SamYaple, redeploying now, | 17:03 |
| inc0 | got some strange errors in the process | 17:03 |
| SamYaple | sdake: you got lucky is all | 17:04 |
| SamYaple | the only reason /run was there in the first place was to share socket with nova-compute | 17:05 |
| *** ftcpops has quit IRC | 17:05 | |
| SamYaple | it just _so_ _happened_ to contain other libvirt stuff too | 17:05 |
| sdake | no it was to share a socket with systemd! | 17:05 |
| sdake | atleast on centos ;) | 17:05 |
| SamYaple | but systemd doesnt use it | 17:05 |
| SamYaple | you are thinking of teh cgroup mount | 17:05 |
| SamYaple | nova-compute used the socket | 17:06 |
| SamYaple | systemd never touched it | 17:06 |
| sdake | ok well whtaever wfm | 17:06 |
| *** daneyon has joined #kolla | 17:06 | |
| sdake | inc0 that patch was already merged | 17:06 |
| *** _ftcpops has joined #kolla | 17:06 | |
| sdake | inc0 so nothign to do there | 17:06 |
| inc0 | yeah, I noticed | 17:06 |
| inc0 | this was today | 17:06 |
| *** daneyon_ has joined #kolla | 17:06 | |
| sdake | inc0 so with that qemu works correctly ? | 17:07 |
| sdake | on container upgrade | 17:07 |
| inc0 | still testing but we have hope | 17:07 |
| sdake | correctly being defined as no downtime or lost state of vm | 17:07 |
| *** gfidente has quit IRC | 17:08 | |
| SamYaple | jeeeeeezzz ugghhh | 17:09 |
| sdake | SamYaple you may be right about systemd + cgroups | 17:10 |
| SamYaple | inc0: finally figured this out | 17:10 |
| inc0 | go ahead | 17:10 |
| SamYaple | sdake: i know i am specifically about cgroups because ubuntu 14.04 doesnt need them | 17:10 |
| SamYaple | 16.04 will though | 17:10 |
| SamYaple | inc0: ok | 17:10 |
| SamYaple | its a PITA | 17:10 |
| SamYaple | and you will confirm it when you show me the output of my requested command | 17:10 |
| SamYaple | inc0: | 17:10 |
| SamYaple | P_INMEM 0x00004 Loaded into memory | 17:10 |
| SamYaple | P_NOCLDSTOP 0x00008 No SIGCHLD when childr | 17:10 |
| *** daneyon has quit IRC | 17:11 | |
| SamYaple | oop wrong two | 17:11 |
| SamYaple | P_CONTROLT 0x00002 Has a controlling terminal | 17:11 |
| SamYaple | P_INMEM 0x00004 Loaded into memory | 17:11 |
| SamYaple | the kvm+qemu process has those two flags set on the process | 17:11 |
| SamYaple | i want to know what _just_ qemu has | 17:11 |
| *** unicell has quit IRC | 17:12 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Switch from mysql-python to pymysql everywhere https://review.openstack.org/276194 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes https://review.openstack.org/275237 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 17:13 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 17:13 |
| SamYaple | hmmmmm inc0 | 17:15 |
| SamYaple | my stuff sticks around whether it is KVM or TCG accel | 17:16 |
| sdake | what is tcg | 17:16 |
| SamYaple | non-kvm | 17:16 |
| sdake | inc0 are you on centos or ubuntu? | 17:16 |
| sdake | SamYaple thanks | 17:16 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 17:17 |
| inc0 | sdake, ubuntu | 17:17 |
| inc0 | nested vm | 17:17 |
| sdake | ok well maybe nested vms just dont work | 17:17 |
| sdake | SamYaple did you try neseted vms? | 17:17 |
| *** fgimenez has quit IRC | 17:17 | |
| SamYaple | inc0: im nested vm | 17:17 |
| SamYaple | im nested kvm and qemu | 17:17 |
| inc0 | but not qemu over xen;) | 17:17 |
| SamYaple | no | 17:17 |
| sdake | kerenels he same? | 17:18 |
| SamYaple | but thats because only RAX uses xen :P | 17:18 |
| inc0 | rax public cloud \o/ | 17:18 |
| inc0 | and amazon | 17:18 |
| SamYaple | true | 17:18 |
| sdake | inc0 your o nrax and amazon? | 17:18 |
| SamYaple | inc0: your kernel is probably 3.13 no? | 17:18 |
| sdake | well christ that is a huge delta | 17:18 |
| inc0 | SamYaple, it's 15.10 so its 4+ | 17:18 |
| SamYaple | 4.2 | 17:18 |
| SamYaple | hmmm yea it should be working | 17:19 |
| inc0 | 4.2.0 yeah | 17:19 |
| sdake | inc0 did you sa you running on rackspaces's cloud? | 17:19 |
| SamYaple | can you walk through your exact proceedure for me? because im all sunshine and daisys over her | 17:19 |
| *** ssurana has joined #kolla | 17:19 | |
| inc0 | yeah | 17:19 |
| sdake | SamYaple are you runnign on your bare metal or rax cloud? | 17:19 |
| inc0 | SamYaple, as soon as I figure out why the hell my kolla is broken | 17:19 |
| SamYaple | i just did a full nova_libvirt NEW IMAGE containerrecrate with upgrade from qemu 2.3 --> 2.5 and it all works | 17:20 |
| inc0 | no downtime? oO | 17:20 |
| SamYaple | i have a 2.3 qemu vm and a 2.5 qemu vm running side by side under nova control | 17:20 |
| SamYaple | no downtime | 17:20 |
| SamYaple | i was ssh'd in the whole time | 17:20 |
| inc0 | O.O | 17:20 |
| sdake | yes this worked in liberty | 17:20 |
| sdake | i spent weeks on it | 17:20 |
| sdake | and it sounds like i t still works | 17:20 |
| sdake | something in inc0's environment is making it not work | 17:20 |
| sdake | if you guys could s ort out what that is that would rock | 17:21 |
| inc0 | n exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 0 anyone seen anything like that? oO | 17:21 |
| sdake | because this problem will be repeated in the field im sure | 17:21 |
| *** mbound has quit IRC | 17:21 | |
| inc0 | http://paste.openstack.org/show/485995/ | 17:22 |
| limamauricio | SamYaple, inc0 I have the same error. ¬¬' | 17:22 |
| limamauricio | ceph aio does not work | 17:23 |
| inc0 | hold on | 17:23 |
| limamauricio | okay | 17:23 |
| limamauricio | :) | 17:23 |
| sdake | limamauricio did yo ufollow directions on setting u p1 osd only? | 17:23 |
| sdake | you only have 1 osd right? | 17:23 |
| limamauricio | I think yes, how i chack it? | 17:24 |
| sdake | moment | 17:24 |
| limamauricio | check* | 17:24 |
| sdake | http://docs.openstack.org/developer/kolla/ceph-guide.html | 17:24 |
| inc0 | limamauricio, ansible --version | 17:25 |
| sdake | [global] | 17:25 |
| sdake | osd pool default size = 1 | 17:25 |
| sdake | osd pool default min size = 1 | 17:25 |
| inc0 | you should have 1.9.1 | 17:25 |
| limamauricio | 1.9.1? | 17:25 |
| limamauricio | I have 1.9.4 | 17:25 |
| inc0 | well less than 2 | 17:25 |
| sdake | thats the version yo uwant to be on | 17:25 |
| sdake | 1.9.4 | 17:26 |
| limamauricio | i'm trying to deploy master | 17:26 |
| inc0 | my mistake | 17:26 |
| sdake | do you have hat global section limamauricio | 17:26 |
| limamauricio | yes | 17:26 |
| limamauricio | :( | 17:26 |
| limamauricio | http://pastebin.com/fFepGZKf | 17:26 |
| limamauricio | ceph_mon logs ^ | 17:26 |
| sdake | limamauricio i have tested ceph all in one in liberty and that worked for me, but I have not tested master ceph aio lately | 17:27 |
| sdake | ceph multinodeworked forme as of two weeks ago | 17:27 |
| SamYaple | limamauricio: you have a _very_ old version of the ceph container | 17:27 |
| inc0 | I did test ceph multinode from master like week ago | 17:27 |
| sdake | my lab is torn apart in a remodel so i can't test it | 17:27 |
| SamYaple | when was the last time you rebuild? | 17:27 |
| SamYaple | please post `docker images` limamauricio | 17:27 |
| limamauricio | ok | 17:27 |
| sdake | limamauricio i am also working on another critical issue here and can't help you live debug atm | 17:27 |
| inc0 | yeah, you should do rebuilt every time you checko out new code | 17:27 |
| inc0 | sdake, did you test this six problem>? | 17:28 |
| sdake | inc0 i had to build a vm | 17:28 |
| sdake | i am getting there | 17:28 |
| inc0 | kk | 17:28 |
| *** neilus has joined #kolla | 17:28 | |
| sdake | getting ready to build containers now | 17:28 |
| limamauricio | http://pastebin.com/2DsqA3B2 | 17:29 |
| limamauricio | http://pastebin.com/Vv64xVku grep ceph | 17:29 |
| limamauricio | ok sdake, thank for your support | 17:30 |
| limamauricio | I'll to deploy multinode | 17:31 |
| SamYaple | limamauricio: you havent rebuild in 2 weeks! | 17:31 |
| SamYaple | you are way outdated | 17:31 |
| limamauricio | :O | 17:31 |
| limamauricio | I have to build the whole week? | 17:31 |
| limamauricio | sorry, but i don't know :/ | 17:32 |
| SamYaple | you have to rebuild everytime somethign changes anythign in docker/* | 17:32 |
| SamYaple | this is master branch, its constantly changing | 17:32 |
| limamauricio | ok | 17:32 |
| limamauricio | ok | 17:32 |
| limamauricio | sorry | 17:32 |
| SamYaple | no problem, but thats the cause of all your problmes | 17:32 |
| limamauricio | i'll run build again :) | 17:32 |
| inc0 | SamYaple, output of ps -p 2666 -o f => F 6 | 17:33 |
| limamauricio | Thank you so much | 17:33 |
| *** neilus has quit IRC | 17:33 | |
| SamYaple | inc0: ok same with me | 17:33 |
| *** akwasnie has quit IRC | 17:33 | |
| SamYaple | im trying to figure out a way to extact KTHREAD info | 17:33 |
| SamYaple | i need to get PF_KTHREAD of the process | 17:34 |
| SamYaple | its flag 0x00200000 | 17:34 |
| inc0 | SamYaple, http://paste.openstack.org/show/486000/ | 17:34 |
| inc0 | vm dies | 17:34 |
| inc0 | Docker version 1.9.1, build a34a1d5 | 17:35 |
| *** athomas has quit IRC | 17:35 | |
| *** salv-orlando has joined #kolla | 17:39 | |
| *** salv-orlando has quit IRC | 17:44 | |
| SamYaple | inc0: I cant figure out this thread thing | 17:45 |
| SamYaple | if my process is a KTHREAD process then we have a thing to test | 17:45 |
| SamYaple | but I can't even figure that our | 17:46 |
| SamYaple | out* | 17:46 |
| SamYaple | (also im not entirely sure on the exact underlying difference) | 17:46 |
| dmsimard | nihilifer: would like to chat with you when you're around - regarding the split of jobs per distro and install type | 17:48 |
| *** unicell has joined #kolla | 17:48 | |
| *** esikachev has joined #kolla | 17:51 | |
| sdake | be back in 1 hour - lunch | 17:56 |
| openstackgerrit | Evgeny Sikachev proposed openstack/kolla-mesos: Add unit-tests for chronos.py https://review.openstack.org/276298 | 17:57 |
| *** daneyon_ has quit IRC | 17:59 | |
| *** Marga_ has quit IRC | 18:05 | |
| *** Marga_ has joined #kolla | 18:05 | |
| pbourke | SamYaple: how do you feel about moving variables from globals.yml to all.yml and leaving globals as all commented out | 18:11 |
| pbourke | SamYaple: for the vars set in globals that aren't in all.yml its impossible to use group_vars/host_vars | 18:11 |
| SamYaple | pbourke: some vars are required in globals.yml | 18:11 |
| SamYaple | most are common options | 18:11 |
| pbourke | true | 18:12 |
| *** sdake has quit IRC | 18:12 | |
| pbourke | network_interface is one that's not though | 18:12 |
| SamYaple | it is a requirement though | 18:12 |
| pbourke | what do you mean its not a requirement | 18:13 |
| SamYaple | it is a requirement | 18:13 |
| pbourke | to have it the same on every node/ | 18:13 |
| pbourke | ? | 18:13 |
| SamYaple | well no but that it needs to be in globals.yml for most installs | 18:15 |
| pbourke | yeah | 18:16 |
| *** salv-orlando has joined #kolla | 18:24 | |
| *** salv-orlando has quit IRC | 18:28 | |
| *** asalkeld has joined #kolla | 18:28 | |
| *** ppetit has quit IRC | 18:29 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges https://review.openstack.org/275573 | 18:30 |
| *** Jeffrey4l has quit IRC | 18:35 | |
| *** Marga_ has quit IRC | 18:38 | |
| *** unicell has quit IRC | 18:46 | |
| *** unicell has joined #kolla | 18:46 | |
| *** neilus has joined #kolla | 18:52 | |
| *** neilus has quit IRC | 18:57 | |
| *** jasonsb has quit IRC | 19:02 | |
| *** esikachev has quit IRC | 19:03 | |
| clayton | SamYaple: do me a favor and run docker run -v /dev:/dev ubuntu and check what /dev/ptmx looks like before and after? | 19:03 |
| clayton | I'd love to just be going crazy. Upgraded a dev environment to docker 1.9.1 and still seeing crazy behavior with ptys | 19:04 |
| *** Marga_ has joined #kolla | 19:04 | |
| *** Marga_ has joined #kolla | 19:04 | |
| *** sdake has joined #kolla | 19:05 | |
| sdake | yo | 19:05 |
| sdake | inc0 did you get that patch tested? | 19:05 |
| sdake | wow house dusty | 19:07 |
| sdake | SamYaple did you get that wiki page setup with the midcycle details? | 19:15 |
| sdake | nm found it | 19:16 |
| *** neilus has joined #kolla | 19:16 | |
| inc0 | sdake, was on lunch | 19:18 |
| sdake | ya me too | 19:18 |
| inc0 | we were debugging libvirt | 19:18 |
| sdake | just got back | 19:18 |
| inc0 | seems like 1.10 doesn't restart vms | 19:18 |
| inc0 | 1.9 does | 19:18 |
| inc0 | seems like a bug tho...fortunate for us but bug nonetheless | 19:18 |
| *** neilus has quit IRC | 19:28 | |
| sdake | samyaple if you ahve the discount link please put in wiki when you have an opportunity | 19:48 |
| sdake | inc0 can you do me a solid and double check this page for errors | 19:48 |
| sdake | https://wiki.openstack.org/wiki/Sprints/KollaMitakaSprint | 19:48 |
| sdake | like in the next 10 minutes | 19:48 |
| sdake | inc0 good news about 1.10 - that is which version we are rolling with | 19:48 |
| inc0 | still confirming but it does seems like it | 19:50 |
| inc0 | as for midcycle site, can't see anything wrong | 19:50 |
| sdake | inc0 can you check that sprint page i want to get it on the ml | 19:50 |
| sdake | there is still a todo for the hotel code | 19:51 |
| sdake | but other then that looks pretty solid | 19:51 |
| inc0 | and an etherpad | 19:51 |
| *** salv-orlando has joined #kolla | 19:54 | |
| inc0 | sdake, I found another bug in stable | 19:56 |
| inc0 | docker-py 1.7 doesn't seem to be working, we need to pin requirements to ==1.1.0 | 19:56 |
| sdake | this is during deploy or buidl? | 19:58 |
| sdake | I am building now with 1.4.0 | 19:58 |
| sdake | so we don't have to pin to 1.1.0 | 19:59 |
| inc0 | deploy | 19:59 |
| inc0 | of stable | 19:59 |
| *** sdake_ has joined #kolla | 20:02 | |
| sdake_ | inc0 i have meeting in 1 hour | 20:03 |
| sdake_ | lets rock | 20:03 |
| *** dratushnyy has quit IRC | 20:03 | |
| sdake_ | let me get my network back | 20:03 |
| inc0 | kk | 20:03 |
| *** sdake has quit IRC | 20:03 | |
| *** sdake has joined #kolla | 20:04 | |
| sdake | inc0 all my containers but gnocchi are built | 20:05 |
| sdake | let me try a deploy | 20:05 |
| inc0 | go for it | 20:05 |
| *** sdake_ has quit IRC | 20:08 | |
| inc0 | so rolling back to docker-py 1.1 and rebuilding container seems to help | 20:10 |
| inc0 | docker-py 1.4 works for you? | 20:10 |
| *** neilus has joined #kolla | 20:17 | |
| *** severion has joined #kolla | 20:18 | |
| SamYaple | clayton: ok give me a minute | 20:26 |
| *** severion has quit IRC | 20:30 | |
| clayton | I suspect your /dev/ptmx is already a symlink | 20:31 |
| SamYaple | clayton: it matches my host perms | 20:31 |
| SamYaple | /dev/ptmx on my host (and in the container) is a dev node | 20:31 |
| SamYaple | not a symlink | 20:31 |
| clayton | you're using docker 1.9.1? | 20:32 |
| SamYaple | yes | 20:32 |
| SamYaple | and docker 1.10 | 20:32 |
| clayton | ok, I can't explain that at all. as near as I can tell libcontainer is hard coded to make that a symlink in the container | 20:33 |
| SamYaple | where do you see that? | 20:33 |
| clayton | https://github.com/docker/docker/blob/master/vendor/src/github.com/opencontainers/runc/libcontainer/rootfs_linux.go#L49-L58 | 20:34 |
| clayton | I compiled a debug version of the docker engine, and it always passes entries in with config.Devices | 20:34 |
| *** fthiagogv_ has joined #kolla | 20:37 | |
| SamYaple | root@y-work-laptop01:~# ls -l /dev/ptmx; docker run --rm -it -v /dev/:/dev/ ubuntu ls -l /dev/ptmx | 20:37 |
| SamYaple | crw-rw-rw- 1 root tty 5, 2 Feb 4 20:36 /dev/ptmx | 20:37 |
| SamYaple | clayton: ^^ | 20:37 |
| SamYaple | crw-rw-rw- 1 root tty 5, 2 Feb 4 20:36 /dev/ptmx | 20:37 |
| clayton | weird, clearly something is different. I'm running of out things it could be | 20:38 |
| SamYaple | whats your host os and kernel? | 20:38 |
| sdake | inc0 i'm getting unrecognized status from pull | 20:38 |
| sdake | I have pull policy set to missing | 20:38 |
| sdake | and am using the registry | 20:38 |
| sdake | I htink the module is not working as expected | 20:38 |
| *** neilus has quit IRC | 20:38 | |
| clayton | Trusty, 3.13.0-67-generic | 20:38 |
| SamYaple | sdake: for liberty? | 20:38 |
| sdake | SamYaple yes | 20:39 |
| sdake | stable/liberty | 20:39 |
| SamYaple | sdake: are you running docker registry v2? | 20:39 |
| sdake | i am not using a registry | 20:39 |
| sdake | i want to just use the cache | 20:39 |
| SamYaple | then you shouldn't be pull an image | 20:39 |
| SamYaple | you likely have your naming wrong | 20:39 |
| SamYaple | you are attempting to pull an image for sure | 20:39 |
| sdake | I see that | 20:39 |
| SamYaple | clayton: just fyi, it is recommened/required to run 3.19 or 4.2 kernel on Trusty | 20:39 |
| sdake | it says "Pullling image docker.io ..." | 20:40 |
| clayton | yeah, I'd like to upgrade, but we're waiting for 16.04. | 20:40 |
| SamYaple | so you likely dont have that image build | 20:40 |
| SamYaple | clayton: you can pull in the latest lts kernel | 20:40 |
| SamYaple | its an official lts kernel in the official repos | 20:40 |
| clayton | sadly there is no newer LTS kernel. There are newer kernels you can run on LTS and they're supported, but only for 6 months | 20:40 |
| SamYaple | there is... | 20:40 |
| SamYaple | no no | 20:41 |
| SamYaple | look at the deprecation map | 20:41 |
| SamYaple | https://wiki.ubuntu.com/Kernel/LTSEnablementStack | 20:41 |
| sdake | openstack-release set improperly samyaple | 20:41 |
| sdake | i'm good | 20:41 |
| SamYaple | if you are saying you must keep kernel for 2 years clayton, then youre right | 20:41 |
| SamYaple | otherwise you ahve many options | 20:41 |
| clayton | SamYaple: yeah, support for all o ftghem but 3.13 expires the day xenial releases | 20:42 |
| sdake | Container command not found or does not exist | 20:42 |
| sdake | never seen that error before | 20:42 |
| sdake | docker 1.10 | 20:42 |
| sdake | docker-py 1.4 | 20:42 |
| SamYaple | clayton: despite what that chart shows, thats not true | 20:42 |
| SamYaple | clayton: you have a 3 month grace period | 20:42 |
| SamYaple | if you look at the super deatailed one toward the bottom it shows that clayton | 20:43 |
| clayton | I can try a newer kernel anyway, it'll be good to rule that out | 20:43 |
| *** macsz1 has joined #kolla | 20:43 | |
| SamYaple | sdake: liberty you need docker 1.8.2 | 20:43 |
| sdake | you said it would work with 1.10 | 20:43 |
| SamYaple | no i said containers will run | 20:43 |
| sdake | oh | 20:44 |
| SamYaple | I said playbooks would not | 20:44 |
| sdake | you didn't say the second part, but I see :) thanks | 20:44 |
| SamYaple | i did repeatedly | 20:44 |
| sdake | good info to have | 20:44 |
| SamYaple | ill get the logs if youd like | 20:44 |
| sdake | well i must have mised it because of these annoyign security rules at cisco | 20:44 |
| sdake | i'll roll back to docker 1.82 | 20:44 |
| *** inc0_ has joined #kolla | 20:44 | |
| SamYaple | its all good | 20:44 |
| SamYaple | you should run your IRC in a vm somewhere | 20:44 |
| SamYaple | to not have that issue | 20:44 |
| clayton | SamYaple: at this point I think it's unlikely we'll find a problem that we'll upgrade for before xenial releases anyway. it's just too close not to wait unless there is a real show stopper | 20:45 |
| SamYaple | I understand clayton, and im not sure that is the issue | 20:46 |
| clayton | SamYaple: reproduced the same issue with 3.19 | 20:46 |
| SamYaple | but just thorwing that out there | 20:46 |
| clayton | yeah, it's worth ruling out | 20:46 |
| clayton | we have some weird aufs stuff we run into that I suspect upgrading would fix also | 20:46 |
| SamYaple | clayton: https://github.com/docker/docker/issues/19464 | 20:46 |
| SamYaple | apparently it is fixed for you in docker 1.10? | 20:47 |
| *** inc0 has quit IRC | 20:47 | |
| *** macsz has quit IRC | 20:47 | |
| *** fthiagogv has quit IRC | 20:47 | |
| clayton | well, I can try 1.10 in a bit | 20:47 |
| SamYaple | clayton: https://github.com/docker/docker/issues/19464 | 20:48 |
| SamYaple | oops | 20:48 |
| clayton | hah, that's a bit of a hack of a fix | 20:48 |
| SamYaple | clayton: https://master.dockerproject.org/ * | 20:48 |
| clayton | I already have the code building locally, it'll be easy to just switch to master and build that | 20:48 |
| clayton | oh, didn't realize those were the binaries, that's definitely easier. thanks | 20:50 |
| SamYaple | yup | 20:50 |
| SamYaple | built per commit too | 20:50 |
| clayton | welll, and I get to inadverently check how long it takes to build the hashes for the local images | 20:51 |
| clayton | SamYaple: yeap, that fixes it | 20:52 |
| SamYaple | clayton: well good news is that will be out in like 3 days | 20:52 |
| SamYaple | it was slated fr a release today | 20:52 |
| SamYaple | and kolla will require it as soon as it lands | 20:53 |
| clayton | SamYaple: I saw something about that, we can wait | 20:53 |
| clayton | so why does this work for you on 1.9.1? | 20:53 |
| SamYaple | im on 15.10 | 20:54 |
| clayton | SamYaple: I think part of this is misconfiguration on ubuntu also. it's not setting ptmxmode when it mounts /dev/pts. if it did then this wouldn't matter | 20:54 |
| clayton | it would change /dev/ptmx to be a symlink to /dev/pts/ptmx and it would still just keep working, since they would have the same permissions | 20:55 |
| clayton | SamYaple: v1.10.0 was tagged 30 minutes ago | 20:56 |
| SamYaple | w00t | 20:56 |
| SamYaple | awesome clayton | 20:56 |
| inc0_ | SamYaple, http://paste.openstack.org/show/486014/ => have you ever seen sth like this? | 20:58 |
| inc0_ | trying to deploy liberty -.- | 20:58 |
| *** neilus has joined #kolla | 21:05 | |
| SamYaple | inc0_: looking | 21:05 |
| sdake | looks like an api change to me | 21:06 |
| sdake | there was something like that in 1.5 of docker-py too | 21:06 |
| sdake | annoying | 21:06 |
| sdake | have to cap docker-py would be my guess | 21:06 |
| inc0_ | I did | 21:06 |
| inc0_ | 1.1.0 | 21:06 |
| sdake | pip show docker-py | 21:07 |
| inc0_ | 1.1.0 | 21:07 |
| SamYaple | inc0_: yea | 21:07 |
| inc0_ | told you, and in requirements we have =>1.1.0 so by default installs latest greatest | 21:07 |
| SamYaple | thats ansible docker module | 21:08 |
| SamYaple | and docker 1.8.2< | 21:08 |
| inc0_ | ahh...I have docker 1.8.2 | 21:08 |
| inc0_ | .3** | 21:08 |
| SamYaple | yup | 21:08 |
| SamYaple | that was the breaking change | 21:08 |
| SamYaple | mid version! | 21:08 |
| sdake | right | 21:08 |
| * sdake wacks docker | 21:08 | |
| * sdake thinks there was an ulterior motive for that nonsense | 21:09 | |
| SamYaple | it was an honest mistake | 21:09 |
| SamYaple | they were solving a different issue | 21:09 |
| SamYaple | it just should have required a bump in x.y.z Y version | 21:09 |
| SamYaple | but they missed it | 21:09 |
| sdake | should hae released 1.8.4 with revert, bumped versions | 21:09 |
| *** absubram has quit IRC | 21:09 | |
| sdake | anyway | 21:10 |
| sdake | can't fix the past :) | 21:10 |
| SamYaple | eh damage was done | 21:10 |
| clayton | SamYaple: just verified, the 1.10 packages are in the apt repos | 21:18 |
| SamYaple | clayton: yup already installed | 21:19 |
| clayton | I'm going to put that into dev and see how it works out | 21:19 |
| SamYaple | testing my needs-docker1.10-patches now | 21:19 |
| SamYaple | about to repush them | 21:19 |
| *** neilus has quit IRC | 21:25 | |
| *** farid_ has quit IRC | 21:27 | |
| *** ssurana has quit IRC | 21:34 | |
| sdake | inc0 when i downgraded to docker 1.8.2 my docker images got removed, so i have to rebuild | 21:35 |
| * sdake groans | 21:35 | |
| sdake | 30-45 mins inc | 21:35 |
| inc0_ | same here | 21:36 |
| inc0_ | just finished;) | 21:36 |
| sdake | i'm just using docker from epel | 21:36 |
| sdake | rather hten docker-engine | 21:36 |
| inc0_ | well, we need to make that crystal clear, that with upgrade you need new docker | 21:37 |
| inc0_ | probably won't be in rh distros yet | 21:37 |
| sdake | inc0_ was that a downgrade from 1.8.3 that caused that? | 21:37 |
| sdake | inc0_ yes i've got an action to write the docs on upgrade | 21:37 |
| sdake | i'll take that on | 21:37 |
| inc0_ | well I had to remove /var/lib/docker to make it work | 21:37 |
| inc0_ | soo...I did succesfully deploy aio stable with docker 1.8.2 | 21:40 |
| SamYaple | inc0_: with my patch in queue? | 21:42 |
| SamYaple | or ithout? | 21:42 |
| inc0_ | vanilla | 21:42 |
| inc0_ | granted I can't run haproxy because it's aio | 21:43 |
| SamYaple | but you an't run deploy with vanilla | 21:43 |
| SamYaple | right? | 21:43 |
| inc0_ | and I think haproxy task was one that bug reported | 21:44 |
| SamYaple | then why did I do this ? https://review.openstack.org/#/c/276328/ | 21:44 |
| inc0_ | I did deploy vanilla | 21:44 |
| inc0_ | just did | 21:44 |
| inc0_ | not with you patch | 21:44 |
| inc0_ | once I solved docker version problem, it deployed | 21:44 |
| inc0_ | people did report bug for that | 21:44 |
| inc0_ | just didnt happen to me | 21:45 |
| *** inc0__ has joined #kolla | 21:46 | |
| inc0__ | irc is fleaky;/ | 21:46 |
| *** inc0__ is now known as inc0 | 21:46 | |
| *** inc0_ has quit IRC | 21:49 | |
| *** sdake has quit IRC | 21:49 | |
| inc0 | sooo...good news is that I kill -9ed docker 1.8.2 and vm keeps runing | 22:02 |
| SamYaple | ahhhhahaharhharharharhhaghaghaghrgahgahg | 22:03 |
| inc0 | bad news is after docker upgrade to 1.10 libvirt containers fails to start | 22:03 |
| SamYaple | docker 1.10 isn't letting me do shared mountpoint | 22:03 |
| inc0 | wait..waht? isn't that exactly why we want docker 1.10? | 22:03 |
| *** neilus has joined #kolla | 22:04 | |
| SamYaple | yea | 22:04 |
| SamYaple | its breaking | 22:04 |
| SamYaple | Error response from daemon: Cannot start container 1642edde5add723fa9253b638e407a679e5e129304cfe17e78f0124e02e04a1f: Path /run/netns is mounted on /run/netns but it is not a shared mount. | 22:04 |
| SamYaple | this is official binaries | 22:04 |
| SamYaple | official 1.10 tagged released bienaries | 22:05 |
| *** absubram has joined #kolla | 22:06 | |
| inc0 | soo upgrading docker leaves you in broken openstack limbo | 22:08 |
| inc0 | that's bad. | 22:08 |
| *** absubram_ has joined #kolla | 22:08 | |
| inc0 | some of containers failed to start | 22:08 |
| SamYaple | inc0: you are stopping the containres first right? | 22:08 |
| *** neilus has quit IRC | 22:08 | |
| SamYaple | then starting them in correct order? | 22:08 |
| inc0 | no | 22:08 |
| inc0 | but that's a good point | 22:09 |
| SamYaple | they have deps on eachother and docker provides no start assurance | 22:09 |
| SamYaple | i think docker starts in order of container id | 22:09 |
| inc0 | however I don't think that's the case | 22:09 |
| inc0 | because one of failing containers is nova_libvirt and it doesn't really have any deps | 22:09 |
| inc0 | and other than that restarting containers that failed should work | 22:10 |
| *** absubram has quit IRC | 22:11 | |
| *** absubram_ is now known as absubram | 22:11 | |
| *** absubram has quit IRC | 22:14 | |
| SamYaple | ahhhhhh | 22:16 |
| SamYaple | got it | 22:16 |
| SamYaple | new default option | 22:16 |
| SamYaple | we need to set system config paramaters in the docker daemon to make this mount sharing work | 22:16 |
| inc0 | make sure to write it down for upgrade-docker playbook | 22:16 |
| SamYaple | im still not sure about that inc0 | 22:17 |
| SamYaple | we dont have a docker-install playbook for a reason | 22:17 |
| SamYaple | why would we have an upgrade one? | 22:18 |
| inc0 | well upgrade will be pretty non-trivial | 22:18 |
| inc0 | even if not as playbook, we need very detailed guide at very least | 22:18 |
| *** neilus has joined #kolla | 22:18 | |
| *** bmace has quit IRC | 22:18 | |
| SamYaple | matters not. the reason we dont have an install playbook is because that is host level | 22:18 |
| SamYaple | detailed guide 100% | 22:18 |
| SamYaple | the playbook is what im on the fence about | 22:18 |
| *** bmace has joined #kolla | 22:19 | |
| SamYaple | i would say we would need an install playbook before upgrade, and we had prevoiusly vetoed the install playbook | 22:19 |
| inc0 | well I think if we keep it completely separated, it might not be too bad | 22:19 |
| SamYaple | see above | 22:19 |
| inc0 | I know | 22:19 |
| SamYaple | ok well thats just a discussion we need to have again | 22:19 |
| inc0 | just my opinion, detailed guide is ok | 22:19 |
| SamYaple | becase it was vetoed since "kolla doesnt tough host" | 22:19 |
| SamYaple | touch* | 22:19 |
| inc0 | yeah I know, but at the same time we want seamless upgrades | 22:20 |
| inc0 | and to peform it as seamless as possible...we need procedure that will be non-trivial at best | 22:21 |
| inc0 | so SamYaple I've been thinking about neutron-agents container rebuild | 22:22 |
| inc0 | we use vxlans as default right? | 22:22 |
| inc0 | then we should be able to migrate between routers pretty easily | 22:22 |
| SamYaple | inc0: we don't migrate anything, the operator does | 22:23 |
| *** neilus has quit IRC | 22:23 | |
| inc0 | again, procedure | 22:23 |
| inc0 | guide, howto | 22:23 |
| SamYaple | vxlan or not, it doesnt matter | 22:23 |
| SamYaple | moving router to new host is same procedure | 22:23 |
| inc0 | if someone uses HA routers, should be pretty easy right? | 22:24 |
| SamYaple | yea that dosent actualy require any action on thier part | 22:24 |
| SamYaple | .... when it works | 22:24 |
| SamYaple | when it doesnt work it still doesnt require action from them | 22:24 |
| SamYaple | but networking drops | 22:24 |
| inc0 | but neutron's problems are neutron's problems | 22:25 |
| SamYaple | yup | 22:25 |
| inc0 | ok | 22:25 |
| SamYaple | to migrate legacy router is pretty simple | 22:25 |
| inc0 | so best case scenerio, we actually can do upgrade with no vm downtime | 22:25 |
| SamYaple | yea | 22:26 |
| SamYaple | network is going down period | 22:26 |
| SamYaple | no getting aroudn that ever | 22:26 |
| SamYaple | until thin containers | 22:26 |
| inc0 | yeah, I get that | 22:26 |
| inc0 | but can we make it a blip? | 22:26 |
| inc0 | not full-fledget, over-10s long downtime? | 22:26 |
| SamYaple | that assumes no issues | 22:26 |
| SamYaple | there are alwasy issues moving routers | 22:26 |
| inc0 | so here's deal, we want to demo upgrades | 22:27 |
| inc0 | so we have benefit of creating configs way we want | 22:27 |
| SamYaple | i would not do live demo if I were you | 22:30 |
| SamYaple | thats just me though | 22:30 |
| SamYaple | it was.... not prepped last time | 22:30 |
| SamYaple | i view that talk as a failure tbh | 22:30 |
| SamYaple | it could have been much better | 22:31 |
| inc0 | I'll run it 100 times before | 22:31 |
| SamYaple | yea | 22:31 |
| SamYaple | still | 22:31 |
| inc0 | live demo adds to "wow, cool" factor tremendously | 22:31 |
| inc0 | but I'll have recording just in case | 22:31 |
| SamYaple | sure | 22:31 |
| inc0 | we'll see, right now I want upgrades at all ;( | 22:32 |
| SamYaple | we should have hard downtime upgrade now and incremental improvements inc0 | 22:33 |
| SamYaple | if those improvments come before mitaka... great! | 22:33 |
| SamYaple | but we need to have _something_ | 22:33 |
| inc0 | so thing is you can do no-downtime-for-vm now | 22:33 |
| inc0 | I did that | 22:33 |
| inc0 | assuming no issues ofc, but yeah, it can be done | 22:34 |
| SamYaple | thats with kill -9 | 22:34 |
| *** tzn has joined #kolla | 22:34 | |
| inc0 | I just want at least that | 22:34 |
| SamYaple | im -2 on automating that | 22:34 |
| inc0 | yes, hacky | 22:34 |
| SamYaple | im ok with documentation | 22:34 |
| inc0 | I'm ok with no automation of that | 22:34 |
| inc0 | just a howto | 22:34 |
| SamYaple | cool | 22:34 |
| inc0 | very detialed | 22:34 |
| SamYaple | yea | 22:34 |
| SamYaple | how and why | 22:34 |
| SamYaple | and why its fixed in 1.11 | 22:34 |
| inc0 | automation will go for openstack services | 22:35 |
| SamYaple | i like how pip now says "still running stup.py for...." | 22:35 |
| SamYaple | its really useful | 22:35 |
| inc0 | that being said, we need to upgrade docker to 1.10 / 1.11 and run liberty on it | 22:35 |
| inc0 | decouple docker upgrade from service upgrade | 22:35 |
| inc0 | yeah, I noticed that too | 22:36 |
| inc0 | so it will probably look like "upgrade docker, then manually start containers in correct order -> here is start_containers_in_correct_order.sh what will do it" | 22:37 |
| inc0 | and you're running liberty on docker 1.11 | 22:37 |
| inc0 | then you run our plays | 22:37 |
| *** tzn has quit IRC | 22:38 | |
| SamYaple | we can backport the kolla_docker module to liberty and _not_ change the plays | 22:38 |
| SamYaple | but add a new play for "start in correct order" | 22:38 |
| inc0 | thing is, it would super cool to run neutron upgrade just once | 22:38 |
| inc0 | we can have it in kolla-mitaka | 22:39 |
| *** salv-orl_ has joined #kolla | 22:39 | |
| inc0 | anyway, that's on me, I'll prep it tomorrow | 22:39 |
| inc0 | as a sh for now, | 22:40 |
| inc0 | I'm going off for now, cyas | 22:40 |
| SamYaple | bye inc0 | 22:40 |
| *** inc0 has quit IRC | 22:40 | |
| *** sdake has joined #kolla | 22:41 | |
| *** salv-orlando has quit IRC | 22:42 | |
| SamYaple | w00000t successful docker 1.10 deploy | 22:42 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 22:46 |
| *** _ftcpops has quit IRC | 22:46 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap https://review.openstack.org/275504 | 22:46 |
| *** jtriley has quit IRC | 22:50 | |
| sdake | SamYaple nice | 22:56 |
| sdake | now if we could just get stable, the thing we released, to work, we woudl be golden ;) | 22:56 |
| *** neilus has joined #kolla | 22:57 | |
| sdake | SamYaple i'm getting an undefined variable attribute ipv4 | 22:58 |
| sdake | seen that one? | 22:59 |
| sdake | ansible 1.9.4-1.el7 | 22:59 |
| *** neilus has quit IRC | 23:01 | |
| sdake | terrible error for a configuration problem | 23:02 |
| sdake | yup well i can duplicate that kolla_ansible problem | 23:05 |
| *** dave-mccowan has quit IRC | 23:05 | |
| sdake | stable/liberty is busted | 23:05 |
| sdake | i'll try your patch now | 23:05 |
| sdake | all day to get a working test environment | 23:05 |
| * sdake groans | 23:05 | |
| *** blahRus has quit IRC | 23:13 | |
| *** pbourke has quit IRC | 23:15 | |
| *** pbourke has joined #kolla | 23:15 | |
| *** JoseMello has quit IRC | 23:18 | |
| *** neilus has joined #kolla | 23:22 | |
| *** neilus has quit IRC | 23:26 | |
| *** aginwala has joined #kolla | 23:27 | |
| *** dims_ has quit IRC | 23:31 | |
| *** dims has joined #kolla | 23:32 | |
| sdake | SamYaple can you ahve a look at that patch you submitted this morning - it is spitting out an oserror 13 related to semaphores | 23:34 |
| sdake | (perm denied) | 23:34 |
| *** neilus has joined #kolla | 23:37 | |
| *** stvnoyes has quit IRC | 23:42 | |
| *** stvnoyes has joined #kolla | 23:43 | |
| *** neilus has quit IRC | 23:48 | |
| *** aginwala has quit IRC | 23:51 | |
| *** aginwala has joined #kolla | 23:56 | |
| *** daneyon has joined #kolla | 23:58 | |
| *** daneyon_ has joined #kolla | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!