| *** vhosakot has quit IRC | 00:01 | |
| *** achanda has quit IRC | 00:03 | |
| *** cfarquhar has joined #kolla | 00:07 | |
| *** cfarquhar has quit IRC | 00:07 | |
| *** cfarquhar has joined #kolla | 00:07 | |
| *** iceyao has joined #kolla | 00:18 | |
| *** iceyao has quit IRC | 00:22 | |
| *** vishwanathj has joined #kolla | 00:27 | |
| *** allen_gao has quit IRC | 00:31 | |
| *** allen_gao has joined #kolla | 00:37 | |
| *** allen_gao has quit IRC | 00:42 | |
| *** vhosakot has joined #kolla | 00:42 | |
| *** allen_gao has joined #kolla | 00:49 | |
| *** allen_gao has quit IRC | 00:54 | |
| *** allen_gao has joined #kolla | 01:01 | |
| *** achanda has joined #kolla | 01:01 | |
| *** vhosakot has quit IRC | 01:09 | |
| *** weiyu_ has joined #kolla | 01:18 | |
| *** achanda has quit IRC | 01:39 | |
| *** cfarquhar has quit IRC | 01:52 | |
| *** allen_gao has quit IRC | 02:07 | |
| *** sdake has joined #kolla | 02:11 | |
| *** allen_gao has joined #kolla | 02:17 | |
| *** sdake has quit IRC | 02:18 | |
| *** achanda has joined #kolla | 02:26 | |
| *** klint has joined #kolla | 02:43 | |
| *** vishwanathj has joined #kolla | 02:48 | |
| *** weiyu_ has quit IRC | 02:55 | |
| *** cfarquhar has joined #kolla | 02:55 | |
| *** cfarquhar has quit IRC | 02:55 | |
| *** cfarquhar has joined #kolla | 02:55 | |
| *** weiyu_ has joined #kolla | 03:02 | |
| openstackgerrit | Allen Gao proposed openstack/kolla-mesos: Fix the deprecated security_group_api and network_api_class https://review.openstack.org/290454 | 03:12 |
|---|---|---|
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr Docker container https://review.openstack.org/298000 | 03:22 |
| *** cbaesema has joined #kolla | 03:23 | |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Use packaged pip instead of easy_install https://review.openstack.org/297523 | 03:25 |
| *** achanda has quit IRC | 03:44 | |
| *** yuanying has quit IRC | 03:53 | |
| *** allen_gao has quit IRC | 03:58 | |
| *** Jeffrey4l has joined #kolla | 04:02 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Implement nova-ssh container https://review.openstack.org/297963 | 04:05 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Restrict the rabbitmq only start on rabbitmq group https://review.openstack.org/297929 | 04:06 |
| *** salv-orlando has joined #kolla | 04:06 | |
| *** allen_gao has joined #kolla | 04:06 | |
| *** salv-orlando has quit IRC | 04:06 | |
| *** salv-orlando has joined #kolla | 04:07 | |
| *** salv-orl_ has quit IRC | 04:09 | |
| *** salv-orlando has quit IRC | 04:12 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Fix gate to use world writeable docker socket https://review.openstack.org/298098 | 04:17 |
| *** coolsvap has joined #kolla | 04:22 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Only copy the necessary config files for Glance https://review.openstack.org/296982 | 04:28 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Refactor the glance service group mapping https://review.openstack.org/298100 | 04:28 |
| *** weiyu_ has quit IRC | 04:30 | |
| *** vishwanathj is now known as vishwanathj_zzz | 04:33 | |
| *** yuanying has joined #kolla | 04:56 | |
| openstackgerrit | Merged openstack/kolla: Use packaged pip instead of easy_install https://review.openstack.org/297523 | 05:00 |
| openstackgerrit | MD NADEEM proposed openstack/kolla: Add optional services option in globals.yml https://review.openstack.org/298104 | 05:27 |
| *** achanda has joined #kolla | 05:31 | |
| *** yuanying has quit IRC | 05:40 | |
| *** yuanying has joined #kolla | 05:48 | |
| *** allen_gao has quit IRC | 05:54 | |
| *** weiyu_ has joined #kolla | 05:59 | |
| *** allen_gao has joined #kolla | 06:04 | |
| *** allen_gao has quit IRC | 06:17 | |
| *** salv-orlando has joined #kolla | 06:21 | |
| *** kjejlly_ has joined #kolla | 06:24 | |
| *** kjelly has quit IRC | 06:25 | |
| *** Marga_ has quit IRC | 06:32 | |
| *** salv-orlando has quit IRC | 06:34 | |
| *** cristicalin has joined #kolla | 06:45 | |
| *** reddy has joined #kolla | 06:49 | |
| *** vishwanathj_zzz has quit IRC | 06:52 | |
| openstackgerrit | Merged openstack/kolla: Add reno support https://review.openstack.org/297959 | 06:56 |
| *** Liuqing has joined #kolla | 06:59 | |
| *** allen_gao has joined #kolla | 07:01 | |
| *** smekel_ has joined #kolla | 07:06 | |
| *** allen_gao has quit IRC | 07:35 | |
| *** phuongnh has joined #kolla | 07:39 | |
| *** allen_gao has joined #kolla | 07:47 | |
| *** cristicalin has quit IRC | 07:47 | |
| *** salv-orlando has joined #kolla | 08:01 | |
| *** salv-orlando has quit IRC | 08:07 | |
| *** stvnoyes has quit IRC | 08:10 | |
| *** stvnoyes has joined #kolla | 08:11 | |
| *** nihilifer has quit IRC | 08:16 | |
| *** nihilifer has joined #kolla | 08:17 | |
| *** reddy has quit IRC | 08:25 | |
| *** yuanying has quit IRC | 08:31 | |
| *** yuanying has joined #kolla | 08:31 | |
| *** yuanying has quit IRC | 08:32 | |
| *** jmccarthy has quit IRC | 08:37 | |
| *** jmccarthy has joined #kolla | 08:38 | |
| *** yuanying has joined #kolla | 08:39 | |
| *** mikelk has joined #kolla | 08:53 | |
| *** mikelk has quit IRC | 08:55 | |
| *** achanda has quit IRC | 09:04 | |
| *** achanda has joined #kolla | 09:11 | |
| *** yuanying has quit IRC | 09:16 | |
| *** kproskurin has joined #kolla | 09:16 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Bump rabbitmq version in rabbitmq-env.conf https://review.openstack.org/298165 | 09:24 |
| *** achanda has quit IRC | 09:28 | |
| openstackgerrit | Merged openstack/kolla-mesos: Fix the deprecated security_group_api and network_api_class https://review.openstack.org/290454 | 09:29 |
| *** kproskurin_ has joined #kolla | 09:33 | |
| *** phuongnh has quit IRC | 09:34 | |
| *** kproskurin has quit IRC | 09:35 | |
| *** kproskurin_ is now known as kproskurin | 09:35 | |
| *** salv-orlando has joined #kolla | 09:36 | |
| *** weiyu__ has joined #kolla | 09:40 | |
| *** weiyu_ has quit IRC | 09:40 | |
| *** weiyu__ has quit IRC | 09:45 | |
| *** salv-orlando has quit IRC | 09:46 | |
| *** weiyu_ has joined #kolla | 09:57 | |
| *** allen_gao has quit IRC | 10:10 | |
| *** allen_gao has joined #kolla | 10:15 | |
| *** pbourke has quit IRC | 10:16 | |
| *** pbourke has joined #kolla | 10:16 | |
| *** Liuqing has quit IRC | 10:25 | |
| *** achanda has joined #kolla | 10:28 | |
| *** achanda has quit IRC | 10:33 | |
| *** mbound has joined #kolla | 10:35 | |
| *** achanda has joined #kolla | 10:36 | |
| *** Liuqing has joined #kolla | 10:51 | |
| *** coolsvap has quit IRC | 10:54 | |
| *** coolsvap has joined #kolla | 11:06 | |
| *** salv-orlando has joined #kolla | 11:11 | |
| *** achanda has quit IRC | 11:11 | |
| *** salv-orlando has quit IRC | 11:13 | |
| *** Liuqing_ has joined #kolla | 11:15 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: iscsi container with lvm2 support https://review.openstack.org/291285 | 11:16 |
| *** Liuqing has quit IRC | 11:16 | |
| *** weiyu_ has quit IRC | 11:24 | |
| *** rhallisey has joined #kolla | 11:25 | |
| *** SiRiuS_ has joined #kolla | 11:35 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Change keystone log dir https://review.openstack.org/297967 | 11:52 |
| *** mlima has joined #kolla | 11:53 | |
| *** Liuqing_ has quit IRC | 12:02 | |
| *** coolsvap has quit IRC | 12:08 | |
| *** achanda has joined #kolla | 12:12 | |
| *** kproskurin has quit IRC | 12:13 | |
| *** ccesario has joined #kolla | 12:14 | |
| *** kproskurin has joined #kolla | 12:14 | |
| kproskurin | Hello guys, I have a question about new keystone bootstrap | 12:14 |
| kproskurin | https://github.com/openstack/kolla/blob/master/docker/keystone/keystone_bootstrap.sh | 12:15 |
| *** mbound has quit IRC | 12:15 | |
| kproskurin | So here we expect vars for admin-url + internal-url + public-url | 12:16 |
| kproskurin | But at current keystone-manage(ver 9.0.0) there is no such parameters | 12:17 |
| kproskurin | So I recive: "keystone-manage: error: unrecognized arguments: --bootstrap-admin-url RegionOne --bootstrap-internal-url --bootstrap-public-url --bootstrap-service-name keystone --bootstrap-region-id" | 12:17 |
| *** achanda has quit IRC | 12:17 | |
| *** mbound has joined #kolla | 12:36 | |
| *** salv-orlando has joined #kolla | 12:50 | |
| *** salv-orlando has quit IRC | 12:53 | |
| *** rhallisey has quit IRC | 12:53 | |
| *** rhallisey has joined #kolla | 12:53 | |
| *** mbound has quit IRC | 12:53 | |
| *** klint has quit IRC | 13:04 | |
| *** vhosakot has joined #kolla | 13:09 | |
| SamYaple | kproskurin youre just using an old version of keystone-manage is all https://github.com/openstack/keystone/blob/9.0.0.0rc1/keystone/cmd/cli.py#L103 | 13:12 |
| kproskurin | SamYaple, strange, it says version 9.0.0 | 13:13 |
| SamYaple | kproskurin: 9.0.0 hasnt been released yet | 13:13 |
| SamYaple | its a dev version for sure | 13:13 |
| SamYaple | lots of times these tools chop off the x.x.xdev34 type strings | 13:14 |
| SamYaple | so it looks like 9.0.0 buti ts not | 13:14 |
| *** weiyu_ has joined #kolla | 13:14 | |
| *** achanda has joined #kolla | 13:15 | |
| kproskurin | SamYaple, oh, ok. Ill check this | 13:15 |
| *** achanda has quit IRC | 13:19 | |
| *** dwalsh has joined #kolla | 13:20 | |
| *** ayoung has joined #kolla | 13:22 | |
| *** vishwanathj_zzz has joined #kolla | 13:22 | |
| *** absubram has quit IRC | 13:24 | |
| *** vhosakot has quit IRC | 13:25 | |
| *** banix has joined #kolla | 13:26 | |
| openstackgerrit | Andrey Pavlov proposed openstack/kolla-mesos: Changing log levels in CLI https://review.openstack.org/298231 | 13:27 |
| *** jtriley has joined #kolla | 13:31 | |
| *** pbourke-home has joined #kolla | 13:43 | |
| *** openstackgerrit has quit IRC | 13:48 | |
| *** openstackgerrit has joined #kolla | 13:48 | |
| *** mbound has joined #kolla | 13:54 | |
| *** mbound has quit IRC | 13:59 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr Docker container https://review.openstack.org/298000 | 14:08 |
| *** jtriley has quit IRC | 14:12 | |
| sbezverk | Good morning | 14:20 |
| sbezverk | SamYaple I see failure of my patch on gate-kolla-docs, when I try to access logs it says file not found? Any ideas why this is happening? | 14:21 |
| *** jtriley has joined #kolla | 14:22 | |
| SamYaple | sbezverk: would need link | 14:22 |
| sbezverk | SamYaple here you go: https://review.openstack.org/#/c/291285/44 | 14:23 |
| patchbot | sbezverk: patch 291285 - kolla - iscsi container with lvm2 support | 14:23 |
| *** vishwana_ has joined #kolla | 14:26 | |
| *** vishwana_ has quit IRC | 14:26 | |
| ccesario | sbezverk, I had the same problem yesterday ..... It seems a gate problem according vhosakot explained me... | 14:28 |
| *** vishwanathj_zzz has quit IRC | 14:29 | |
| *** vishwanathj has joined #kolla | 14:29 | |
| sbezverk | ccesario thank you for letting me know, I was scratching my head trying to figure out how I managed to screw up docu :-) | 14:30 |
| ccesario | sbezverk, eheheheh, maybe SamYaple can confirm it :) | 14:31 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add Kuryr Docker container https://review.openstack.org/298000 | 14:31 |
| *** sdake has joined #kolla | 14:42 | |
| *** rstarmer has quit IRC | 14:46 | |
| *** sdake has quit IRC | 14:46 | |
| *** sdake has joined #kolla | 14:47 | |
| *** JoseMello has joined #kolla | 15:01 | |
| *** pbourke-home has quit IRC | 15:03 | |
| *** inc0 has joined #kolla | 15:04 | |
| *** ph03n1x has joined #kolla | 15:08 | |
| *** weiyu_ has quit IRC | 15:09 | |
| *** Marga_ has joined #kolla | 15:12 | |
| *** Marga_ has quit IRC | 15:13 | |
| *** Marga_ has joined #kolla | 15:13 | |
| *** ssurana has joined #kolla | 15:14 | |
| *** achanda has joined #kolla | 15:17 | |
| inc0 | good morning | 15:18 |
| -openstackstatus- NOTICE: Gerrit is restarting on review.openstack.org in an attempt to address an issue reading an object from the ec2-api repository | 15:20 | |
| *** achanda has quit IRC | 15:22 | |
| *** absubram has joined #kolla | 15:23 | |
| *** blahRus has joined #kolla | 15:28 | |
| *** salv-orlando has joined #kolla | 15:32 | |
| Marga_ | Hi! | 15:34 |
| *** salv-orlando has quit IRC | 15:41 | |
| *** mdnadeem_home has joined #kolla | 15:48 | |
| *** vhosakot has joined #kolla | 15:49 | |
| *** ph03n1x has quit IRC | 15:53 | |
| *** Serlex has joined #kolla | 16:00 | |
| *** mdnadeem_home has quit IRC | 16:01 | |
| *** pbourke-home has joined #kolla | 16:03 | |
| *** pbourke-home has quit IRC | 16:05 | |
| mlima | helo guys, i have this error | 16:06 |
| mlima | TASK: [mariadb | Running MariaDB bootstrap container | 16:06 |
| mlima | full log -> http://pastebin.com/5428NGhx | 16:07 |
| mlima | aio, enable_haproxy: "yes" | 16:07 |
| inc0 | mlima, "address already in use" | 16:08 |
| inc0 | do you have mysql running on host already? | 16:08 |
| inc0 | netstat -nalp | grep 3306 | 16:09 |
| inc0 | or if you are running haproxy | 16:09 |
| inc0 | what kolla_external_addess do you have? | 16:09 |
| inc0 | is it unused addr from your network? | 16:09 |
| sdake | morning folks | 16:10 |
| mlima | hmm, i didn't change it | 16:10 |
| mlima | thank you inc0 | 16:10 |
| inc0 | yeah, that's it, haproxy bound to 3306 | 16:11 |
| inc0 | you can either turn it off for aio or give non-used address and keepalived will manage it | 16:11 |
| inc0 | this way haproxy will bind to this addr:3306 and host-ip:3306 mariadb will take | 16:12 |
| sdake | kolla_external_address doesn't nee to be changed if you leave it commented out | 16:12 |
| sdake | kolla will dot he right thing, only kolla_internal_address really needs to be set | 16:12 |
| inc0 | well it would be good to have addr from your network | 16:12 |
| inc0 | otherwise multinode will not work | 16:13 |
| sdake | kolla_external_address = kolla_internal_address if fkolla_external-address is unchanged | 16:13 |
| inc0 | ok, you need to set one of them, thats what I'm saying | 16:13 |
| inc0 | but yeah, sdake is right, kolla_internal_addess is more important | 16:13 |
| *** Marga_ has quit IRC | 16:13 | |
| mlima | :) | 16:13 |
| sdake | if yous set kolla external address and dont set kolla internal address I think kolla wouldbust | 16:13 |
| *** ph03n1x has joined #kolla | 16:14 | |
| inc0 | yeah, that's true, my mistake | 16:14 |
| inc0 | but you get the idea mlima :) | 16:14 |
| mlima | yes, inc0 :) | 16:14 |
| vhosakot | ccesario: I tested the patch set in https://review.openstack.org/#/c/297967/4 and see that it does not work.. I have provided my comments in the review... let me know if you ahve any questions | 16:15 |
| patchbot | vhosakot: patch 297967 - kolla - Change keystone log dir | 16:15 |
| manjeets | i updated the kolla repo and it works with mariadb . older which version which pip gets does not works with mariadb | 16:20 |
| manjeets | without removing two patches in mariadb | 16:21 |
| manjeets | and now I am getting failure at very end soon will get the logs and post currently a run is going on | 16:21 |
| ccesario | vhosakot, thaks by comments....... but the current code already create the correct directory..... https://github.com/openstack/kolla/blob/master/docker/keystone/extend_start.sh#L15-L25 or I 'm wrong ? | 16:22 |
| inc0 | manjeets, what's the failure? | 16:22 |
| manjeets | just getting to that re ran the deploy | 16:22 |
| inc0 | was that on neutron deploy? | 16:22 |
| manjeets | no on cinder | 16:22 |
| inc0 | oh, ok so something else | 16:23 |
| vhosakot | ccesario: that should create.. but the code is not hitting that part in the shell script... let me update to master and re-try... could you test as well if you get a chanbc | 16:23 |
| inc0 | keep in mind that cinder is fleaky in containers | 16:23 |
| inc0 | that's why we suggest to use ceph underneath | 16:23 |
| ccesario | vhosakot, I have tested before ask review | 16:23 |
| manjeets | it will take probably 10 mins to get that failure | 16:23 |
| ccesario | the all logs are in /var/log/kolla/keystone/ | 16:24 |
| manjeets | I saw neutron containers and nova got done suucessfully but while doing cinde it was ending up in error | 16:24 |
| vhosakot | nice... give me around 20 mins | 16:24 |
| ccesario | vhosakot, sure. | 16:24 |
| manjeets | inc0: ok I don't need cinder for my case if that did not work i will disable cinder | 16:24 |
| manjeets | go with nova and neutron | 16:24 |
| sdake | manjeets my bro | 16:25 |
| inc0 | manjeets, well it should deploy tho | 16:25 |
| sdake | manjeets i need some benchmarks out of your 64 node cluster when you have an opportunity | 16:25 |
| manjeets | okay it takes approax 23 minutes to finish run for those nodes | 16:26 |
| sdake | to deploy 3 controlller nodes? | 16:26 |
| manjeets | trying to get time on every run | 16:26 |
| sdake | and the compute nodes | 16:26 |
| manjeets | no all 64 nodes | 16:27 |
| sdake | you deployed controller on 64 nodes? | 16:27 |
| manjeets | yes it was ending in issue for storage have not got that completely yet | 16:27 |
| manjeets | another run is in process | 16:27 |
| manjeets | will post the issue | 16:28 |
| sdake | controllers should be 3 or max 5 | 16:28 |
| sdake | compute andn storage shold e the remainder | 16:28 |
| inc0 | well you dont need storage for cinder-less stiuff | 16:28 |
| inc0 | what I do on this kind of hardware is I use every compute node as ceph osd | 16:29 |
| inc0 | which translates to every compute node being storage node | 16:29 |
| manjeets | so we can't have more than 5 control nodes ? | 16:31 |
| sdake | you can | 16:31 |
| sdake | i just dont recommend it ;) | 16:32 |
| inc0 | manjeets, there is no need for 5 controllers | 16:32 |
| sdake | manjeets i want benchmarks on upgrade and reconfigure | 16:32 |
| inc0 | waste of resources | 16:32 |
| sdake | for upgrade, I need to show you how to generate an upgrade evnrionment probably via google hangouts | 16:32 |
| manjeets | inc0: ok | 16:32 |
| sdake | i'd like the results for 3 controller nodes, and the remainder compute or storage nodes as needed | 16:33 |
| manjeets | what is threshold for network nodes ? | 16:33 |
| manjeets | all i care is scaling and performance of network nodes | 16:33 |
| *** SiRiuS_ has quit IRC | 16:38 | |
| manjeets | inc0: i see two failure today | 16:39 |
| manjeets | http://paste.openstack.org/show/492106/ | 16:39 |
| manjeets | http://paste.openstack.org/show/492107/ | 16:40 |
| sdake | manjeets i also need you to use the overlay graph driver for your deployment | 16:41 |
| manjeets | overlay graph driver ? | 16:41 |
| sdake | -s overlay flag to docker | 16:41 |
| sdake | rm -rf /var/lib/docker | 16:42 |
| sdake | sudo systemctl stop docker first | 16:42 |
| sdake | the nrm | 16:42 |
| sdake | the nstart it up again ater adding -s overlay flag | 16:42 |
| sdake | dont use it on your build nodes | 16:42 |
| sdake | only your deploy targets | 16:42 |
| sdake | it should be faster | 16:42 |
| sdake | so if you have 1-2 hours to go over what i want in terms of benchmarks today or osme otherday let me know | 16:43 |
| SamYaple | /msg/win 11 | 16:43 |
| SamYaple | morning | 16:43 |
| sdake | yo SamYaple | 16:43 |
| sdake | SamYaple manjeets ha a 64 node deployment - 23 minutes to deploy | 16:44 |
| sdake | and he is deploying contorller nodes on 64 nodes lol | 16:44 |
| inc0 | sdake, no, hes not | 16:44 |
| inc0 | he's doing 7 controllers | 16:44 |
| sdake | ok | 16:44 |
| inc0 | but still too much | 16:45 |
| sdake | he said above he wa doing 64 controllers | 16:45 |
| sdake | but when i looked at his environment previously he was doing 7 | 16:45 |
| manjeets | inc0: will chqange it | 16:45 |
| inc0 | you misunerstood each other | 16:45 |
| manjeets | to 3 or 5 | 16:45 |
| sdake | 5 gives you super ha | 16:45 |
| sdake | 3 givees you ha ;) | 16:45 |
| manjeets | okay | 16:45 |
| inc0 | manjeets, yeah that should be enough...these are damn powerful machines | 16:45 |
| inc0 | sdake, not really....it don't really matter that much 3v5 | 16:45 |
| inc0 | doesn't | 16:46 |
| manjeets | can i have more network nodes ? | 16:46 |
| manjeets | what is recommendation for network > | 16:46 |
| inc0 | manjeets, 2 network nodes is enough, it's active-passive anyway | 16:46 |
| manjeets | my problem is to check the scale and network performance mainly | 16:46 |
| inc0 | with dvr? | 16:46 |
| inc0 | or without? | 16:46 |
| manjeets | with dvr | 16:46 |
| inc0 | so dvr's mostly reside on compute nodes | 16:47 |
| inc0 | number of network nodes won't greatly increase network throughput | 16:47 |
| inc0 | also...vlans or vxlans? | 16:47 |
| sdake | sbezverk irc dude ;) | 16:49 |
| sdake | sbezverk i'll eview that iscsi patch when i hit the reviw queue this morning | 16:49 |
| manjeets | vxlans | 16:49 |
| manjeets | with ovs and linuxbridge separately | 16:50 |
| inc0 | try both;) | 16:50 |
| inc0 | well, bottom line | 16:50 |
| manjeets | so here is my new set of deployment | 16:50 |
| manjeets | 3 control | 16:50 |
| inc0 | 2-3 net nodes should be enough | 16:50 |
| manjeets | 3 network | 16:50 |
| manjeets | 50 computes | 16:50 |
| manjeets | rest storage | 16:50 |
| manjeets | ? | 16:50 |
| sdake | manjeets can you do that with overlayfs | 16:50 |
| manjeets | sure | 16:51 |
| inc0 | you don't need stor at all without cinder or swift | 16:51 |
| inc0 | so I'd say put all the rest to compute | 16:51 |
| sdake | manjeets lets setup a google hangout so i can make sure ou do it correctly plz :) | 16:51 |
| sdake | its not documented really anywhere | 16:51 |
| manjeets | inc0: ok just doing prechecks and cleanup | 16:52 |
| manjeets | sdake when you will be free ? | 16:52 |
| *** unicell has quit IRC | 16:53 | |
| gmmaha | sdake: inc0: I and ntpttr were just trying a deploy and were wondering how you guys feel about making pre-check as the first step of deploy always? | 16:54 |
| gmmaha | so that if somethign is off, we dont even try to deploy but fix the hosts before the deploy actually happens | 16:54 |
| sdake | whisky tango foxtrot http://logs.openstack.org/98/298098/1/check/gate-kolla-dsvm-deploy-centos-binary/4f8dc1a/console.html#_2016-03-28_07_10_43_381 | 16:54 |
| sdake | gmmaha it takes too long imo | 16:55 |
| inc0 | gmmaha, we do something like this in rabbitmq hosts | 16:55 |
| inc0 | I'm actually good with that | 16:55 |
| inc0 | but not as first step, but nova prechecks before nova deploy and such | 16:55 |
| gmmaha | sdake: true.. we add some overhead, but thought its better than half deploy to find something is wrong, then clean the containers & hosts, run cehck, fix and then start again! | 16:56 |
| gmmaha | :) | 16:56 |
| inc0 | also I started to make post-deploy tests so it will stop deployment if for example rabbit doesn't work after deploy | 16:56 |
| gmmaha | inc0: just stop when that piece fails? | 16:56 |
| inc0 | gmmaha, yeah, pretty much | 16:56 |
| gmmaha | inc0: hmm.. so lets say it happens and fails.. whats the next steps? | 16:57 |
| sdake | oh nm that i the stable branch | 16:57 |
| gmmaha | Fix that piece and restart the play just from that spot? | 16:57 |
| inc0 | if you fail on preckeck, you will get clear message what's wrong | 16:57 |
| inc0 | go and debug;) | 16:57 |
| gmmaha | inc0: absolutely | 16:57 |
| gmmaha | but what if i didnt run pre-check and deploy fails somehow | 16:57 |
| gmmaha | for obvious reasons | 16:57 |
| gmmaha | just thtought that pre-emptively assujming that the deployer has a non-perfect state of hosts to deploy and running pre-check would be good for us | 16:58 |
| inc0 | I agree with you | 16:58 |
| inc0 | I'm ok with adding minute or two more to deploy | 16:58 |
| inc0 | it is rocket fast anyway;) | 16:58 |
| inc0 | and we can optimize | 16:59 |
| inc0 | bottom line, this is good place to do some refactoring | 16:59 |
| inc0 | because we do some precheck-style stuff in deploy as well | 16:59 |
| sdake | can we finish mitaka first please ;) | 16:59 |
| gmmaha | inc0: nice.. yeah agree with you on that | 16:59 |
| manjeets | inc0: for me prechecks all green not error | 16:59 |
| gmmaha | sdake: sorry didnt mean to digress a current plan.. Just thought it will be a great wish-list item | 16:59 |
| inc0 | good for N | 17:00 |
| inc0 | we need to do some refactoring anyway | 17:00 |
| manjeets | i am starting a run 3 control 3 networks and rest compute | 17:00 |
| inc0 | as we move to ansible 2.0 | 17:00 |
| sdake | manjeets with overlayfs? | 17:00 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Remove bootstrap_mariadb with cleanup-containers https://review.openstack.org/298328 | 17:00 |
| manjeets | yes i am doing that change as well | 17:00 |
| sdake | remember to stop the daemon | 17:01 |
| sdake | rm -rf /var/lib/kolla | 17:01 |
| sdake | systemctl daemon-reload | 17:01 |
| sdake | rather /var/lib/docker | 17:01 |
| *** salv-orlando has joined #kolla | 17:02 | |
| *** Marga_ has joined #kolla | 17:02 | |
| sdake | bugs need fixing: | 17:03 |
| sdake | https://launchpad.net/kolla/+milestone/mitaka-rc3 | 17:03 |
| manjeets | sdake service docker restart -s ? | 17:03 |
| manjeets | start** | 17:03 |
| sdake | i dont know what -s flag does to restart | 17:03 |
| manjeets | no start | 17:03 |
| sdake | the way you can tell if overlay is being used is via p -ef | grep docker | 17:03 |
| sdake | you have to stop docker daemon | 17:03 |
| sdake | you have to rm -r /var/lib/docker | 17:03 |
| sdake | then you have to start docker daemon | 17:04 |
| sdake | *ONLY ON YOUR DEPLOY TARGETS* | 17:04 |
| sdake | inbetween the stop and start you need to add -s overlay to the start options | 17:04 |
| sdake | in docker | 17:04 |
| sdake | i am not quite sure hwo to do this on debuntian | 17:04 |
| *** cbaesema has quit IRC | 17:05 | |
| *** ssurana has quit IRC | 17:07 | |
| manjeets | i ran docker daemon -storage-driver=overlay | 17:09 |
| *** ssurana has joined #kolla | 17:10 | |
| sdake | -s overlay | 17:11 |
| sdake | no equal sign i think | 17:12 |
| manjeets | sadake docker daemon --storage-driver=overlay | 17:12 |
| manjeets | is taking time | 17:12 |
| sdake | i think you dont need the equal sign | 17:12 |
| manjeets | showing something like http://paste.openstack.org/show/492116/ | 17:13 |
| sdake | oh your on ubuntu | 17:13 |
| * sdake groans | 17:13 | |
| manjeets | how long it takes ? | 17:13 |
| sdake | inc0 SamYaple do either off you knwo which ubuntu kernel is required to run with overlay? | 17:14 |
| inc0 | well we suggest to run wily kernel anyway | 17:14 |
| manjeets | I have wily | 17:14 |
| manjeets | which did on all nodes | 17:14 |
| sdake | does that support overlay? | 17:14 |
| inc0 | if it's newer that wily (I doubt that) than I suggest to not run it | 17:14 |
| manjeets | i don't think so it is | 17:14 |
| inc0 | you may need to install different packages tho | 17:14 |
| sdake | uese the flag -s overlay | 17:14 |
| manjeets | is overlay fs is must or can i go without it ? | 17:15 |
| sdake | i have also heard your filesystem must be ext4 | 17:15 |
| sdake | manjeets i want to see if fit prodcues a speed delta | 17:15 |
| inc0 | manjeets, it's completely optional | 17:15 |
| sdake | people are speed freaks around deployment | 17:15 |
| inc0 | may make build and deploy quicker | 17:15 |
| *** salv-orlando has quit IRC | 17:16 | |
| manjeets | inc0 what packages are required for this ? | 17:16 |
| sdake | inc0 i asked him to see what the delta is for deploy | 17:16 |
| inc0 | manjeets, not sure...I use aufs;) | 17:16 |
| manjeets | its heading no where just stuck saying that old message | 17:16 |
| inc0 | old habits die hard;) | 17:16 |
| sdake | manjeets pleae use -s overlay | 17:16 |
| sdake | not --storage-ddriver=overlay | 17:16 |
| manjeets | ok | 17:16 |
| sdake | also paste your /etc/fstab | 17:17 |
| manjeets | i saw --storage-driver = overlay on docker doc | 17:17 |
| manjeets | sdake same thing happens i don't see any difference | 17:17 |
| manjeets | docker daemon -s overlay | 17:17 |
| sdake | http://askubuntu.com/questions/109413/how-do-i-use-overlayfs | 17:18 |
| *** unicell has joined #kolla | 17:19 | |
| daneyon | Any known issues running Kolla on Ubuntu 14.04.3 LTS (GNU/Linux 3.16.0-30-generic x86_64) | 17:19 |
| sdake | https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/ | 17:19 |
| *** unicell has quit IRC | 17:19 | |
| *** unicell1 has joined #kolla | 17:19 | |
| inc0 | daneyon, for once we suggest in docs to upgrade kernel to wily | 17:20 |
| vhosakot | daneyon: share /run.... mount --make-shared /run in http://docs.openstack.org/developer/kolla/quickstart.html | 17:20 |
| inc0 | stuff break without new kernel | 17:20 |
| daneyon | Should I follw http://docs.openstack.org/developer/kolla/quickstart.html instead of https://github.com/openstack/kolla/blob/master/doc/quickstart.rst | 17:21 |
| inc0 | follow one on master | 17:22 |
| manjeets | inc0: any way to check which storage driver is currenlt being run ? | 17:22 |
| *** achanda has joined #kolla | 17:22 | |
| inc0 | not sure if theyre not the same | 17:22 |
| inc0 | manjeets, so if you didn't change anything default is devicemapper I think | 17:22 |
| inc0 | which isn't pretty | 17:22 |
| inc0 | if you change to aufs, which is just an option change, deploy should speed up considerably | 17:23 |
| manjeets | I am trying docker daemon -s AUFS | 17:23 |
| vhosakot | daneyon: I think both are same | 17:24 |
| vhosakot | yep, refer master | 17:24 |
| inc0 | overlay or btrfs is even better performance-wise, but it's not as dramatic change | 17:24 |
| vhosakot | ccesario: I am rebuilding heka now | 17:24 |
| manjeets | how to switch to that ? | 17:25 |
| manjeets | i tried using docker daemon -s but now its freezing whenever i use docker command | 17:26 |
| inc0 | vim /etc/default/docker | 17:26 |
| manjeets | that just has DOCKER_OPTS | 17:27 |
| *** achanda has quit IRC | 17:27 | |
| daneyon | vvhosakot: OK, thx. | 17:27 |
| vhosakot | cool | 17:27 |
| inc0 | it's a docker_opts entry:) | 17:27 |
| inc0 | add -s aufs | 17:27 |
| *** SamYaple has left #kolla | 17:27 | |
| inc0 | and service docker restart | 17:27 |
| manjeets | okay | 17:27 |
| vhosakot | manjeets: "docker info | grep Storage" shows what storage driver the daemon is currently running | 17:28 |
| manjeets | I already have aufs | 17:30 |
| sdake | we want -s overlay ;) | 17:30 |
| manjeets | sdake let me have one run | 17:32 |
| manjeets | i will do overlay later | 17:32 |
| sdake | manjeets ok as long as you getme data - i'll love you long time ;) | 17:33 |
| manjeets | lol | 17:33 |
| *** mark-casey has joined #kolla | 17:36 | |
| ntpttr | hey kolla people, gmmaha and I are having some trouble with libvirt. we cleaned up our hosts and made sure libvirt was stopped, passed prechecks, but on deployment libvirt doesn't start | 17:39 |
| ntpttr | we get these logs http://paste.openstack.org/show/492120/ | 17:39 |
| ntpttr | anyone know what might be the issue? | 17:40 |
| mark-casey | hello. I was wondering if the image build process builds any secrets into the images such that the internal docker registry could become an attractive attack target, or if that kind of thing (perhaps credentials for inter-service interactions) passed at runtime? | 17:41 |
| inc0 | ntpttr, looks like libvirtd is broken...logs plz | 17:41 |
| *** SiRiuS_ has joined #kolla | 17:42 | |
| ntpttr | inc0: here's libvirtd.log http://paste.openstack.org/show/492121/ | 17:43 |
| ntpttr | power management can't be found? | 17:43 |
| inc0 | strange | 17:43 |
| ntpttr | pm-is-supported | 17:43 |
| inc0 | sudo apt-get -y install pm-utils try to do it in container | 17:44 |
| ntpttr | the internet says "install pm-utils will slove the problem." | 17:44 |
| inc0 | and restart | 17:44 |
| ntpttr | ah yep | 17:44 |
| *** Marga_ has quit IRC | 17:44 | |
| ntpttr | inc0: in nova_libvirt container? or host? | 17:45 |
| gmmaha | inc0: since the container is on a constant restart path, how can we install it inside the container? | 17:45 |
| inc0 | nova_libvirt | 17:46 |
| *** Marga_ has joined #kolla | 17:46 | |
| inc0 | try to add this line to dockerfile and rebuild then | 17:46 |
| *** achanda has joined #kolla | 17:46 | |
| inc0 | if it fixes the issue, publish a patch plz:) | 17:46 |
| ntpttr | inc0: where do the dockerfiles live? | 17:46 |
| sdake | ntpttr the docker dir | 17:47 |
| sdake | under nova | 17:47 |
| sdake | under libvirt | 17:47 |
| ntpttr | ah just in the kolla repo kk | 17:47 |
| inc0 | then kolla build and deploy | 17:47 |
| ntpttr | there's already a line in that dockerfile to install pm-utils | 17:48 |
| sdake | libvirtd has a connection refused | 17:48 |
| *** ph03n1x has quit IRC | 17:48 | |
| sdake | is libvirtd in a restart loop? | 17:48 |
| ntpttr | sdake: don't think so | 17:49 |
| sdake | docker ps -a | grep libvirt | 17:50 |
| sdake | 2016-03-28 17:37:37.207 1 ERROR nova.virt.libvirt.host libvirtError: unable to connect to server at '172.16.2.6:16509': Connection refused | 17:50 |
| ntpttr | sdake: oh the container, all of the nova containers are in a restart loop | 17:50 |
| ntpttr | sorry, yes | 17:50 |
| sdake | docker logs nova_libvirt -> paste | 17:50 |
| ntpttr | sdake: here it is, looks like a permissions issue http://paste.openstack.org/show/492125/ | 17:52 |
| mlima | sdake, i did a script to clean fstab | 17:52 |
| mlima | :) | 17:52 |
| mlima | I clean all references to ceph | 17:53 |
| ntpttr | sdake: I'm running everything as root though | 17:53 |
| *** ph03n1x has joined #kolla | 17:54 | |
| inc0 | ntpttr, you might have to, or at least allow libvirt to do stuff | 17:54 |
| inc0 | remember that it plays with kernel | 17:54 |
| ntpttr | inc0: I already am | 17:54 |
| inc0 | ahh | 17:54 |
| inc0 | drop-root gone bad sdake? | 17:54 |
| sdake | mlima nice | 17:55 |
| *** mark-casey1 has joined #kolla | 17:59 | |
| *** mark-casey has quit IRC | 18:01 | |
| *** Jeffrey4l has quit IRC | 18:04 | |
| *** unicell1 has quit IRC | 18:07 | |
| *** unicell has joined #kolla | 18:07 | |
| vhosakot | I see nova_libvirt is run as root.. I see all nova containers except nova_libvirt are run as nova user, and nova_libvirt is run as root.... "docker exec nova_libvirt whoami" shows root.. | 18:09 |
| ntpttr | vhosakot: think I should change that setting to fix my permissions error? | 18:11 |
| manjeets | sdake, inc0 success on kolla http://paste.openstack.org/show/492129/ | 18:11 |
| manjeets | this is with all images already built | 18:11 |
| vhosakot | ntpttr: kolla already runs nova_libvirt as root | 18:11 |
| inc0 | 11min - 64 nodes? | 18:11 |
| sdake | manjeets were your nodes cleaned up of images? | 18:11 |
| manjeets | inc0: this is with all the images already on it | 18:11 |
| manjeets | no | 18:11 |
| manjeets | i did not cleanup images to save time | 18:11 |
| sdake | can you do that | 18:11 |
| vhosakot | ntpttr: can you paste the output of grep USER /usr/local/share/kolla/docker/nova/nova-libvirt/Dockerfile.j2 | 18:11 |
| inc0 | well build images is ok | 18:11 |
| ntpttr | vhosakot: no output from that command | 18:12 |
| vhosakot | ntpttr: that means, no nova USER for nova_libvirt... can you paste the output from other nova container like nova_compute ? grep USER /home/kolla/kolla/docker/nova/nova-compute/Dockerfile.j2 | 18:13 |
| manjeets | what is domain in horizon saw it for first time | 18:14 |
| manjeets | ? | 18:14 |
| ntpttr | vhosakot: that one has user nova | 18:14 |
| manjeets | while logging horizon dashboard | 18:14 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add a step to clean all ceph references in fstab https://review.openstack.org/298363 | 18:14 |
| vhosakot | ntpttr: I dont think it is user issue in nova_libvirt container.. | 18:15 |
| ntpttr | vhosakot: hmm no clue what's causing it now, this was working just last week | 18:16 |
| *** salv-orlando has joined #kolla | 18:18 | |
| manjeets | inc0: what is domain on horizon board ? | 18:18 |
| manjeets | never saw it before | 18:18 |
| inc0 | default | 18:18 |
| manjeets | asking me to put domain, usename and password | 18:18 |
| inc0 | this is new in KS | 18:18 |
| inc0 | I think | 18:18 |
| inc0 | just type in "default" | 18:18 |
| manjeets | cool | 18:18 |
| manjeets | thansk | 18:19 |
| *** JoseMello has quit IRC | 18:19 | |
| sdake | domain is default | 18:20 |
| ccesario | vhosakot, got success!? | 18:20 |
| vhosakot | ccesario: will keep you posted | 18:24 |
| *** salv-orlando has quit IRC | 18:25 | |
| ccesario | right... | 18:26 |
| *** sdake_ has joined #kolla | 18:34 | |
| *** sdake_ has quit IRC | 18:35 | |
| *** sdake has quit IRC | 18:37 | |
| *** SiRiuS_ has quit IRC | 18:37 | |
| *** sdake has joined #kolla | 18:39 | |
| vhosakot | ntpttr: don't know... would be a good check to see the permissions of libvirt-admin.so in /var/lib/docker.... | 18:39 |
| vhosakot | ntpttr: I meant checking the permission of each file that shows up in the output of sudo find /var/lib/docker | grep 'libvirt-admin.so$' | 18:39 |
| *** banix_ has joined #kolla | 18:42 | |
| vhosakot | ccesario: tested, works, gave +1, nice work! | 18:43 |
| *** banix has quit IRC | 18:43 | |
| *** banix_ is now known as banix | 18:43 | |
| *** sdake has quit IRC | 18:44 | |
| gmmaha | vhosakot: inc0: should this still be there? https://github.com/openstack/kolla/blob/master/ansible/roles/horizon/templates/local_settings.j2#L158 | 18:44 |
| gmmaha | it seems that the altest keystone donesnt need it and this is causing the issue where we cannot create new roles/projects without creating the _member_ role | 18:45 |
| vhosakot | gmmaha: if you need _member_, you'll need that | 18:45 |
| gmmaha | vhosakot: right, but any reason we should enable it by default? | 18:46 |
| vhosakot | gmmaha: are you using the "openstack project/user/role create" command ? | 18:46 |
| gmmaha | vhosakot: have tried both the commadn line and the horizon dashboard | 18:47 |
| gmmaha | i think both failed with the same error | 18:47 |
| * gmmaha isnt sure about the command line | 18:47 | |
| gmmaha | i am 100% sure that the horizon dashboard failed | 18:47 |
| manjeets | what password is used when you try to do something as sudo on container | 18:48 |
| *** sdake has joined #kolla | 18:48 | |
| manjeets | i am trying to into namespace inside neutron_l3_agent container | 18:48 |
| manjeets | it is asking password for neutron | 18:48 |
| vhosakot | gmmaha: hmmm... ok... I can successfully create a user without _member_ and "openstack role create test" works fine for me | 18:48 |
| gmmaha | vhosakot: aaah | 18:48 |
| gmmaha | then its just horizon that will crap out! | 18:48 |
| manjeets | tried host and neutron_keystone password from password,yml | 18:48 |
| gmmaha | and thats bad isnt it? | 18:48 |
| sdake | inc0 what do you have on your plate atm that needs reallocation now that your taking over the 1.1.0 backports | 18:49 |
| vhosakot | gmmaha: well, horizon will see the user _not_ with _member_ role.. yes.... | 18:49 |
| gmmaha | vhosakot: right and it will deny creating of any new user/project until i get the _member_ role inclued/added | 18:50 |
| vhosakot | manjeets: docker exec -tu root <container name> works for me | 18:50 |
| inc0 | sdake, I was pretty free after rabbitmq stuff, wanted to fix this bug: https://bugs.launchpad.net/kolla/+bug/1546789 | 18:50 |
| openstack | Launchpad bug 1546789 in kolla "neutron failed to deploy in a multi-node deployment" [High,Triaged] | 18:50 |
| gmmaha | vhosakot: how do you feel about a patch that will comment out that line? | 18:50 |
| * gmmaha was going to ask that | 18:50 | |
| inc0 | I'll be happy if someone takes care about it | 18:50 |
| inc0 | this one is painful | 18:50 |
| ccesario | vhosakot, thanks ;) | 18:51 |
| vhosakot | gmmaha: yes, but I think creating user/role/project is operator task after kolla is dpeloyed.. so, I'm thinking should kolla do it... would be a good Q to the entire team | 18:51 |
| vhosakot | ccesario: no, nice work | 18:51 |
| *** achanda has quit IRC | 18:51 | |
| vhosakot | no=np | 18:51 |
| gmmaha | vhosakot: absolutely.. but expecting the operator to create _member_ when there is no need for it.. thats what i was curious about.. cause keystone doesnt expect you to need that role to anything | 18:52 |
| gmmaha | vhosakot: thanks for the comments.. will wait for others' comments on it.. | 18:53 |
| *** SiRiuS_ has joined #kolla | 18:53 | |
| sdake | inc0 the workaround in #7 no bueno? | 18:53 |
| * manjeets created a instance that got two ip from local network | 18:53 | |
| vhosakot | gmmaha: yep... I will dig the code a bit and get back to you... cool | 18:54 |
| inc0 | sdake, I wanted to have this workaround in master | 18:54 |
| inc0 | it will make things better | 18:54 |
| *** achanda has joined #kolla | 18:54 | |
| sdake | inc0 how many nodes it require to duplicate? | 18:54 |
| gmmaha | vhosakot: thanks a lot | 18:54 |
| inc0 | so it happened on 7 for me | 18:54 |
| sdake | nobody in the comunity has this many nodes | 18:55 |
| sdake | would it happen with 3 nodes? | 18:55 |
| inc0 | can't say, if you make separated controller from compute it might | 18:55 |
| vhosakot | gmmaha: the gate uses the default user/role/domain/project and it works fine.. I have been able to boot VMs fine with the default user/role/domain/project... | 18:55 |
| inc0 | I suspect it's because of enable_manila | bool | 18:55 |
| vhosakot | gmmaha: so, I will dig the code a bit and see if kolla must create the _member_ role for the operator | 18:56 |
| gmmaha | vhosakot: i am guessing both the gate and your tests are using command line tools? | 18:56 |
| vhosakot | yep.. | 18:56 |
| vhosakot | openstack CLI | 18:56 |
| gmmaha | vhosakot: per the folks at keystone we dont need it anymore.. | 18:56 |
| sdake | inc0 looks like an esy workaround - not sur how much work in testing | 18:56 |
| gmmaha | which is why the command line works (I guess) | 18:56 |
| gmmaha | so horizon setting tht as the default role is throwing a monkey wrench when you try creating users/projects using GUI | 18:57 |
| gmmaha | so unless horizon needs it, i think its fair to throw it out | 18:57 |
| inc0 | sdake, yeah, but it did break for me and it's hard to pinpoint if you're not know what to look at | 18:57 |
| sdake | inc0 that workaorund broke? | 18:57 |
| sdake | in comment #7? | 18:57 |
| inc0 | no, workaround worked | 18:57 |
| inc0 | for me | 18:57 |
| sdake | ok well git review it and lets moce on ;) | 18:57 |
| inc0 | but since it's not invasive | 18:58 |
| inc0 | I was thinking about putting it in master | 18:58 |
| inc0 | I dont have code now;) | 18:58 |
| sdake | it needs t obe backported too | 18:58 |
| inc0 | yeah | 18:58 |
| inc0 | or no | 18:58 |
| inc0 | actually, it might not | 18:58 |
| inc0 | because we don't have manila in Lib | 18:58 |
| vhosakot | gmmaha: kolla-toolbox setups KS https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/kolla_keystone_user.py#L62-L78 | 18:58 |
| sdake | i mean backported to mitaka | 18:58 |
| inc0 | yeah, that's correct | 18:59 |
| gmmaha | vhosakot: hmmm.. | 19:00 |
| gmmaha | it definitely doesnt setup _member_ in my deploys.. | 19:00 |
| gmmaha | can you check if that role exists in your setup? | 19:00 |
| sdake | inc0 since its such a simple fix and you expressed interest in fixing it, i dont think we need to clear it for you doing 1.1.0 backports | 19:00 |
| vhosakot | gmmaha: but the backend KS code that runs is the same for all clients - CLI, python client, horizon | 19:00 |
| sdake | so I assigned it toyou sa a result :) | 19:00 |
| vhosakot | gmmaha: let me check.. 1 min | 19:00 |
| inc0 | sdake, yeah | 19:01 |
| inc0 | I'll just do it | 19:01 |
| gmmaha | vhosakot: right the keystone is the same | 19:01 |
| *** mark-casey has joined #kolla | 19:03 | |
| *** dwalsh has quit IRC | 19:03 | |
| gmmaha | vhosakot: https://github.com/openstack/horizon/blob/711f88dd4785fdb3cd26bb0d7b8ddf38037f5b8d/openstack_dashboard/dashboards/identity/projects/workflows.py#L200-L216 | 19:03 |
| *** dwalsh has joined #kolla | 19:04 | |
| *** mark-casey1 has quit IRC | 19:04 | |
| gmmaha | and with that we are getting poushed out by horizon from creating others even though keystone doesnt care/need you to have it | 19:04 |
| vhosakot | gmmaha: aha :) | 19:04 |
| *** cbaesema has joined #kolla | 19:06 | |
| gmmaha | inc0: sdake: same error with re-build of the images and re-deploy http://paste.openstack.org/show/492140/ | 19:06 |
| gmmaha | :( | 19:06 |
| vhosakot | gmmaha: as you pointed we alread have OPENSTACK_KEYSTONE_DEFAULT_ROLE=_member_ in https://github.com/openstack/kolla/blob/master/ansible/roles/horizon/templates/local_settings.j2#L158... but KS is not pikcing it up | 19:06 |
| sdake | inc0 sdake is the same person apparently ;) | 19:07 |
| gmmaha | sdake: :) | 19:07 |
| sdake | gmmaha are you on ubutu or centos | 19:07 |
| gmmaha | sdake: Ubuntu | 19:08 |
| sdake | gmmaha do following | 19:09 |
| sdake | sudo -i | 19:09 |
| sdake | cd /var/lib/kolla | 19:09 |
| sdake | find . -name libvirt-admin.so.0 -ls | 19:09 |
| sdake | rather cd /var/lib/docker | 19:09 |
| gmmaha | sdake: guessing this is in the container? | 19:09 |
| sdake | when you find the file | 19:09 |
| sdake | ls -l it to see its permissions | 19:09 |
| gmmaha | sdake: http://paste.openstack.org/show/492142/ | 19:10 |
| *** dwalsh has quit IRC | 19:11 | |
| sdake | ls -l the file its linked to plz | 19:11 |
| *** dwalsh has joined #kolla | 19:11 | |
| gmmaha | sdake: http://paste.openstack.org/show/492144/ | 19:12 |
| sdake | docker exec nova_libvirt id | 19:12 |
| gmmaha | sdake: Error response from daemon: Container nova_libvirt is restarting, wait until the container is running | 19:13 |
| vhosakot | gmmaha: does the horizon error say "OPENSTACK_KEYSTONE_DEFAULT_ROLE" is missing ? | 19:13 |
| sdake | oh is this horizon? | 19:13 |
| gmmaha | vhosakot: right.. it will say we cannot cerate the user | 19:13 |
| sdake | I thought wew ereon libvirt | 19:13 |
| gmmaha | sdake: two parallel things :) | 19:13 |
| gmmaha | vhosakot has been helping me with an interesting issue with horizon and keystone roles | 19:14 |
| sdake | gmmaha ok lets stay focused on libvirt for the moment | 19:14 |
| gmmaha | sdake: you got it | 19:14 |
| vhosakot | gmmaha: sdake: wont mix things.. you guys carry on.. I want to check something in KS users in kolla | 19:14 |
| vhosakot | go shead | 19:14 |
| sdake | cd /var/lib/docker | 19:15 |
| sdake | find . -name libvirtd.log | 19:15 |
| sdake | paste the contents of that file | 19:15 |
| gmmaha | sdake: nothing.. no file with that name exists | 19:15 |
| sdake | moment let me read the code | 19:16 |
| sdake | try libvirt.log | 19:16 |
| gmmaha | sdake: is that in /var/lib/docker or /var/log/libvirt(local host) | 19:16 |
| sdake | /var/lib/docker | 19:17 |
| gmmaha | sdake: no file exists with name libvirt*log | 19:17 |
| sdake | docker logs nov_libvirt | 19:18 |
| sdake | paste | 19:18 |
| sdake | plz | 19:18 |
| sbezverk | sdake lately I noticed permission denied when accessing libvirtd.log | 19:18 |
| sbezverk | sdake it is probably runs as root and heka used does not have access there.. | 19:18 |
| gmmaha | sdake: http://paste.openstack.org/show/492146/ | 19:19 |
| sdake | gmmaha run docker --version | 19:20 |
| gmmaha | sdake: Docker version 1.10.3, build 20f81dd | 19:20 |
| sdake | gmmaha run ps -ef | grep docker | 19:20 |
| gmmaha | sdake: root 2231 1 0 16:23 ? 00:00:48 /usr/bin/docker daemon --insecure-registry 172.16.2.2:4000 | 19:21 |
| sdake | was it working prior and stopped working? | 19:21 |
| sdake | this looks like some kind of wierd docker bug to me | 19:21 |
| gmmaha | sdake: yeah it used to work all last week | 19:22 |
| sdake | and youd id what to braek it? :) | 19:22 |
| gmmaha | we did a fresh deploy on friday to switch from OVS to linuxbridge and noticed that we no longer see compute hosts on horizon dashboard | 19:22 |
| sdake | by fresh deploy did you cleanup-images? | 19:24 |
| sdake | and cleanup-containers? | 19:24 |
| sdake | and rebuild | 19:24 |
| sdake | or what | 19:24 |
| sdake | define "fresh deploy" plz :) | 19:25 |
| gmmaha | sdake: full remove of imges, re-buld and redeploy | 19:25 |
| gmmaha | with cleanup of containers and hosts | 19:25 |
| sdake | how old were your previous images? | 19:26 |
| gmmaha | barely a week old | 19:26 |
| sdake | run docker images | grep libvirt | 19:26 |
| gmmaha | 172.16.2.2:4000/kollaglue/ubuntu-source-nova-libvirt 2.0.0 3cd0d481d32b 11 days ago 630.7 MB | 19:27 |
| sdake | did you run cleanup-images? | 19:27 |
| gmmaha | hm m | 19:27 |
| gmmaha | i thought we did | 19:27 |
| gmmaha | ntpttr: ^^ | 19:27 |
| sdake | i dont know what is causing the problem | 19:27 |
| ntpttr | gmmaha: sdake: yeah we ran cleanup-images | 19:27 |
| sdake | i dont know if the linuxbride container was tested | 19:28 |
| sdake | when it was made into a thin container | 19:28 |
| sdake | here is what I recommend | 19:28 |
| sdake | run cleanup-containers and cleanup-images on every host | 19:28 |
| sdake | reboot every hot | 19:28 |
| sdake | revert your ovs to linuxbrige config optoin change | 19:28 |
| sdake | and try a fresh deploy with your images in your registry | 19:28 |
| sdake | in the meantime i am going to try a linuxbridge deployment on my side | 19:34 |
| sdake | because my environment definately works ;) | 19:34 |
| *** vhosakot has quit IRC | 19:34 | |
| *** vhosakot has joined #kolla | 19:35 | |
| *** sdake_ has joined #kolla | 19:36 | |
| *** sdake has quit IRC | 19:38 | |
| *** salv-orlando has joined #kolla | 19:42 | |
| sdake_ | ntpttr in the midst of deploying linuxbridge, i'll let you know how that goes | 19:46 |
| sdake_ | manjeets around? | 19:47 |
| ntpttr | sdake_: thanks, we're rebuilding images now | 19:47 |
| sdake_ | ntpttr i didn't really want you to rebuild images but ok ;) | 19:48 |
| ntpttr | sdake_: oh you just wanted us to clean them up on all the target hosts? | 19:48 |
| ntpttr | though that was implied with the cleanup-images | 19:48 |
| sdake_ | yup | 19:48 |
| sdake_ | cleanup-images removes the images from the docker image cache | 19:49 |
| sdake_ | but not from the registry | 19:49 |
| ntpttr | sdake_: oh so building them again is creating duplicates in the registry? or does it just overwrite? | 19:50 |
| *** salv-orlando has quit IRC | 19:52 | |
| sdake_ | it overwrites if you use the same tag number | 19:54 |
| sdake_ | i change my tag number for every build i do | 19:54 |
| sdake_ | so i can go back in time and change things around ;) | 19:54 |
| sdake_ | (2)No such file or directory: AH02291: Cannot access directory '/var/log/kolla/horizon/' for error log of vhost defined at /etc/httpd/conf.d/horizon.conf:3 | 19:54 |
| sdake_ | vhosakot horizon not working | 19:54 |
| sdake_ | did you do the horizon container log change? | 19:54 |
| sdake_ | ntpttr i can't get a vm to boot in nova for some reason with or without linuxbridge, but linuxbridge doesnt' caue libvirtd to restart loop | 19:56 |
| sbezverk | vhosakot there were two patches for horizon, both must be present for horizon logging to work.. | 19:56 |
| ntpttr | sdake_: weird we were able to boot instances before this issue stopped libvirt from working | 19:56 |
| sdake_ | ntpttr it may just be a problem on centos | 19:57 |
| sdake_ | i am not certain yet | 19:57 |
| manjeets | sdake yes | 19:57 |
| sdake_ | manjeets did you get overlayfs working with docker? | 19:58 |
| manjeets | not yet having rabbit mq issues | 19:58 |
| manjeets | i get 2 ip's when i spin a instance | 19:58 |
| manjeets | when i do docker exec -tu root rabbitmq bash | 19:59 |
| *** Serlex1 has joined #kolla | 19:59 | |
| manjeets | and it just freezes | 19:59 |
| manjeets | i need to watch logs which needs sudo access | 19:59 |
| manjeets | and password for rabbitmq in password.yml is not working | 19:59 |
| manjeets | vhosakot: here ? | 20:01 |
| sdake_ | docker exec heka tail -f /var/log/rabbitmq/rabbitmq.log | 20:01 |
| manjeets | ok | 20:01 |
| sdake_ | rather /var/log/kolla/rabitmq/rabbitmq.log | 20:01 |
| *** Serlex has quit IRC | 20:02 | |
| *** vhosakot has quit IRC | 20:02 | |
| *** sdake has joined #kolla | 20:03 | |
| sdake | sbezverk where is the other patch for horizon -> link plz | 20:05 |
| *** sdake_ has quit IRC | 20:06 | |
| *** pbourke-home has joined #kolla | 20:08 | |
| sbezverk | sdake_ https://bugs.launchpad.net/kolla/+bug/1561073 | 20:08 |
| openstack | Launchpad bug 1561073 in kolla "incomplete horizon logging fix" [Undecided,Fix released] - Assigned to Serguei Bezverkhi (sbezverk) | 20:08 |
| *** vhosakot has joined #kolla | 20:09 | |
| *** mark-casey1 has joined #kolla | 20:09 | |
| vhosakot | sdake: yes.. saw the IRC logs.. file bug for horizon | 20:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add creation of new log folder for Horizon https://review.openstack.org/298418 | 20:09 |
| *** pbourke-home has quit IRC | 20:09 | |
| sdake | ^^needs cr approvals | 20:09 |
| vhosakot | ah you submitted the patch.. cool | 20:10 |
| *** akscram has quit IRC | 20:11 | |
| *** mark-casey has quit IRC | 20:11 | |
| *** akscram has joined #kolla | 20:11 | |
| *** mark-casey1 has left #kolla | 20:12 | |
| sdake | i just ued the cherrypick button | 20:14 |
| sdake | it was already in master | 20:14 |
| sdake | i have stable/mitaka checked out | 20:14 |
| sdake | manjeets after you finish the job on overlay benchmarking i'd like to know what a tls enabled registry produces in terms of performance deficits ;) | 20:15 |
| sdake | and then upgrade and then reconfigure | 20:15 |
| sdake | and i'll be done with my benchmark requets :) | 20:16 |
| vhosakot | manjeets: I just checked.... rabbitmq in passwords/yml works fine for me in lastet master | 20:34 |
| *** vhosakot has quit IRC | 20:34 | |
| *** vhosakot has joined #kolla | 20:36 | |
| manjeets | ok latest master ? | 20:36 |
| sdake | manjeets if your using master you have to use samyaple's generate-password.py tool | 20:36 |
| vhosakot | manjeets: yes... | 20:36 |
| vhosakot | yes, kolla-genpwd... | 20:36 |
| sdake | this will filll out the password file | 20:36 |
| sdake | tools/generate-password.py is the link | 20:36 |
| sdake | then go look in the password file for the passwords ;) | 20:36 |
| ntpttr | sdake: we finished the reinstall and still libvirt is broken | 20:37 |
| sdake | did you rebuild images? | 20:37 |
| ntpttr | sdake: yeah | 20:37 |
| sdake | are you using master? | 20:37 |
| ntpttr | sdake: I havent done a pip install on master | 20:38 |
| ntpttr | we're using the same version that worked last week | 20:38 |
| sdake | so you haven't updated your repos? | 20:38 |
| ntpttr | I think we did, but didn't pip install kolla | 20:38 |
| sdake | you dont want to pip install kolla | 20:38 |
| ntpttr | right gmmaha, you did update the repo? | 20:38 |
| sdake | that needs to go from the docs | 20:38 |
| sdake | perhaps a recent change broke libvirt | 20:39 |
| sdake | let me try master in my environment moment | 20:39 |
| gmmaha | ntpttr: yes i updated the code base to the latest master | 20:39 |
| vhosakot | gmmaha: which same error ? permission issue on libvirt-admin.so ? or, pm-is-supported not found error ? | 20:43 |
| *** sdake_ has joined #kolla | 20:43 | |
| ntpttr | sdake_: we've got to go and do some other stuff for now, but will be back to this later | 20:43 |
| *** sdake has quit IRC | 20:43 | |
| sdake_ | ntpttr it takes me about 15 mins to buidl and 5 mins to deploy i'll leave a note if master is working | 20:43 |
| ntpttr | sdake_: awesome, thank you | 20:44 |
| gmmaha | vhosakot: yeah.. think the same error.. | 20:45 |
| gmmaha | i am stuck with soemthing for a bit | 20:45 |
| gmmaha | i will update it asap | 20:45 |
| gmmaha | when i am out of there | 20:45 |
| gmmaha | here* | 20:45 |
| vhosakot | gmmaha: also, which of the two error ? :) | 20:45 |
| vhosakot | or both | 20:45 |
| sdake_ | how do folks feel about the ansible offices is nc for the next midcycle? | 20:47 |
| sdake_ | or maybe its sc ;) | 20:47 |
| vhosakot | did you mean RH offices ? :) | 20:47 |
| sdake_ | ansible has their own office | 20:47 |
| sdake_ | but ya, its a red hat facility | 20:48 |
| vhosakot | ah ok | 20:48 |
| vhosakot | cool | 20:48 |
| gmmaha | vhosakot: http://paste.openstack.org/show/492160/ | 20:49 |
| *** banix has quit IRC | 20:49 | |
| vhosakot | hmm.. same permission issue for libvirt-admin.so.. ok | 20:51 |
| *** achanda has quit IRC | 20:52 | |
| vhosakot | gmmaha: can you paste the output of "ps aux | grep libvirtd"... | 20:53 |
| gmmaha | vhosakot: on the host i presume? | 20:56 |
| vhosakot | yes.. compute host | 20:56 |
| gmmaha | vhosakot: nothing but the grep command itself | 20:56 |
| vhosakot | ah... ok.. 1 sec | 20:56 |
| vhosakot | gmmaha: your nova_libvirt container is not up as well, right ? | 21:01 |
| gmmaha | vhosakot: thats the one thats going on restarting | 21:01 |
| gmmaha | vhosakot: yes its not up and running | 21:01 |
| *** jtriley_ has joined #kolla | 21:01 | |
| *** achanda has joined #kolla | 21:04 | |
| *** jtriley has quit IRC | 21:04 | |
| *** Marga_ has quit IRC | 21:06 | |
| *** jtriley_ has quit IRC | 21:06 | |
| *** Marga_ has joined #kolla | 21:06 | |
| vhosakot | gmmaha: your permissions match mine for libvirt in http://paste.openstack.org/show/492142/ | 21:10 |
| vhosakot | gmmaha: can you paste the output of "sudo find /var/lib/docker -name libvirt-admin.so.0.1003.1 | xargs sudo ls -l" | 21:10 |
| vhosakot | I have it -rw-r--r-- | 21:11 |
| gmmaha | vhosakot: i have the smae | 21:11 |
| gmmaha | sme* | 21:11 |
| gmmaha | same* | 21:11 |
| vhosakot | -rw-r--r-- ? | 21:11 |
| gmmaha | -rw-r--r-- 1 root root 30680 Mar 2 04:20 /var/lib/docker/aufs/diff/1c9abd1375465d1015865015e8cb8c09f0b4a4caafe4826d4261b92336162fb4/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0.1003.1 | 21:11 |
| gmmaha | -rw-r--r-- 1 root root 30680 Mar 2 04:20 /var/lib/docker/aufs/diff/48c7f01c7d48558ff82b836c4c1534b5917d1c6978e717ffe07de4feb5f67c28/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0.1003.1 | 21:11 |
| gmmaha | vhosakot: yeah | 21:12 |
| sdake_ | gmmaha my libvirt on masater with linuxbridge doens't enter a recycle loop | 21:15 |
| gmmaha | sdake_: aaah | 21:15 |
| gmmaha | then definitely somethign wrong with the hosti guess.. | 21:15 |
| gmmaha | :( | 21:15 |
| gmmaha | i can try and redeploy the base OS on the compute hosts and start all over again! | 21:15 |
| sdake_ | do you know how to turn off that thing that is ike selinux in ubuntu | 21:15 |
| sdake_ | apparmor | 21:16 |
| sdake_ | gmmaha ls -l the symlinks again plz | 21:16 |
| gmmaha | sdake_: hmm.. you think selinux persmissions are mucking with this? | 21:16 |
| gmmaha | sdake_: sure | 21:16 |
| sdake_ | ubuntu doesn't have selinux | 21:16 |
| gmmaha | -rw-r--r-- 1 root root 30680 Mar 2 04:20 ./aufs/diff/1c9abd1375465d1015865015e8cb8c09f0b4a4caafe4826d4261b92336162fb4/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0.1003.1 | 21:17 |
| gmmaha | -rw-r--r-- 1 root root 30680 Mar 2 04:20 ./aufs/diff/48c7f01c7d48558ff82b836c4c1534b5917d1c6978e717ffe07de4feb5f67c28/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0.1003.1 | 21:17 |
| vhosakot | gmmaha's symlinks' permission match with my env in which I have a booted a working vm fine | 21:17 |
| sdake_ | no, the symlinks | 21:17 |
| sdake_ | not what they link to | 21:17 |
| gmmaha | vhosakot: sdake_: the same env worked last week | 21:17 |
| gmmaha | its starting today that its acting up | 21:17 |
| gmmaha | sdake_: aaah | 21:17 |
| vhosakot | sdake_: I see gmmaha already sent that.. is this what you're looking for ? http://paste.openstack.org/show/492142/ | 21:18 |
| gmmaha | lrwxrwxrwx 1 root root 25 Mar 2 04:20 ./aufs/diff/1c9abd1375465d1015865015e8cb8c09f0b4a4caafe4826d4261b92336162fb4/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0 -> libvirt-admin.so.0.1003.1 | 21:18 |
| gmmaha | lrwxrwxrwx 1 root root 25 Mar 2 04:20 ./aufs/diff/48c7f01c7d48558ff82b836c4c1534b5917d1c6978e717ffe07de4feb5f67c28/usr/lib/x86_64-linux-gnu/libvirt-admin.so.0 -> libvirt-admin.so.0.1003.1 | 21:18 |
| vhosakot | gmmaha: there is a way to start libvirtd in debug mode.. 1 sec | 21:19 |
| sdake_ | i have no libvirt-admin in centos | 21:20 |
| sdake_ | /var/lib/docker/overlay/fbc95f417506830845bd6921820d26fdf5e6e38dd7d4eb3ee2a894b3e6512636/root/usr/lib64/libvirt-admin.so.0 | 21:21 |
| sdake_ | ok i have idea | 21:22 |
| sdake_ | find the libvort .so 1003 file | 21:22 |
| sdake_ | and do the following to it | 21:22 |
| sdake_ | chmod 755 the file | 21:22 |
| gmmaha | sdake_: clean containers after that and redeploy? | 21:22 |
| sdake_ | -rwxr-xr-x. 3 root root 28144 Feb 16 12:09 /var/lib/docker/overlay/fdf4439f1831cccbdd1e051fa1a5c3b58d4760d6a1b2062f41628785d13c8903/root/usr/lib64/libvirt-admin.so.0.1002.17 | 21:23 |
| sdake_ | no | 21:23 |
| sdake_ | just change the permissions | 21:23 |
| gmmaha | sdake_: done | 21:23 |
| sdake_ | still looping? | 21:23 |
| vhosakot | sdake_: gmmaha already has -rw-r--r-- for libvirt-admin.so.0.1003.1 | 21:23 |
| sdake_ | ya my file for that has execute bit set | 21:23 |
| gmmaha | sdake_: let me set it | 21:24 |
| vhosakot | sdake_: ah... change it to 755.... ah | 21:24 |
| vhosakot | I have -rw-r--r-- ... only read bits set | 21:24 |
| sdake_ | and it works on your gear hosakot? | 21:24 |
| vhosakot | yep, both OVS and LinuxBrdige... AIO.. I just told gmmaha that | 21:25 |
| *** sdake_ is now known as sdake | 21:27 | |
| vhosakot | gmmaha: let us know what happens after you retest with 755... I just found out the way to start libvirtd in debug mode | 21:27 |
| gmmaha | vhosakot: sure | 21:27 |
| *** salv-orlando has joined #kolla | 21:27 | |
| gmmaha | vhosakot: sdake: didnt work.. the perms change | 21:29 |
| vhosakot | gmmaha: ok... do this | 21:29 |
| sdake | jpeeler rhallisey ping | 21:30 |
| vhosakot | gmmaha: in https://github.com/openstack/kolla/blob/master/ansible/roles/nova/templates/nova-libvirt.json.j2#L2 change "command": "/usr/sbin/libvirtd --listen", to "command": "/usr/sbin/libvirtd -v --listen", and redeploy nova_libvirt | 21:30 |
| rhallisey | sdake, yo | 21:31 |
| sdake | you will hae to remove the contanier not just redeploy because a restarting container isn't redeployed by our software atm unfortunately | 21:31 |
| vhosakot | gmmaha: add -v | 21:31 |
| gmmaha | vhosakot: aah | 21:31 |
| vhosakot | gmmaha: there is more.. 1 sec | 21:31 |
| gmmaha | vhosakot: sue | 21:31 |
| gmmaha | sure* | 21:31 |
| vhosakot | gmmaha: you need to do the change in /usr/local/share/kolla/ansible/roles/nova/templates/nova-libvirt.json.j2 and re-deploy nova_libvirt | 21:33 |
| sdake | cleanup-containers nova -f | 21:33 |
| sdake | or nova_libirt | 21:34 |
| sdake | not sure exact syntax | 21:34 |
| sdake | you dont want that livirt container looping while you redeploy | 21:34 |
| vhosakot | gmmaha: and then pastebin the output of "docker exec -u root heka cat /var/log/kolla/libvirt/libvirtd.log" | 21:34 |
| *** dwalsh has quit IRC | 21:35 | |
| vhosakot | gmmaha: yep, dont re-deploy a broken looping container... stop it, remove it, make the changes in nova-libvirt.json.j2, redeploy | 21:35 |
| *** salv-orlando has quit IRC | 21:35 | |
| vhosakot | kolla-ansible deploy -t nova | 21:35 |
| gmmaha | vhosakot: sdake: will do it soon.. off to some meetings.. :( | 21:37 |
| * gmmaha hates mondays filled with meetings | 21:37 | |
| sdake | we hae meeting free monday at cisco | 21:37 |
| sdake | except as my dirctor informs me, it doesn't apply to principals ;) | 21:37 |
| * gmmaha is jealous | 21:37 | |
| *** achanda has quit IRC | 21:37 | |
| vhosakot | haha | 21:37 |
| gmmaha | sdake: :) | 21:37 |
| gmmaha | sdake: i have to rebuild the docker images? | 21:40 |
| gmmaha | or just destroy and deploy is god? | 21:41 |
| gmmaha | good* | 21:41 |
| *** Serlex1 has quit IRC | 21:44 | |
| *** salv-orlando has joined #kolla | 21:45 | |
| *** cfarquhar has quit IRC | 21:45 | |
| sdake | just redpeloy gmmaha | 21:45 |
| gmmaha | vhosakot: the file doesnt exist in the heka container | 21:46 |
| gmmaha | cat: /var/log/kolla/libvirt/libvirtd.log: No such file or directory | 21:46 |
| vhosakot | it should.... the code latest ? | 21:47 |
| gmmaha | vhosakot: as of this afternoon.. | 21:47 |
| gmmaha | i did a 'git pull' in the kolla project and re-built them all | 21:47 |
| vhosakot | gmmaha: can you do docker exec -u root heka find / -name libvirtd.log | 21:47 |
| *** cfarquhar has joined #kolla | 21:48 | |
| *** cfarquhar has quit IRC | 21:48 | |
| *** cfarquhar has joined #kolla | 21:48 | |
| gmmaha | vhosakot: empty | 21:48 |
| gmmaha | no files found | 21:48 |
| vhosakot | gmmaha: did you add -v and re-deploy ? | 21:48 |
| gmmaha | vhosakot: yeah | 21:48 |
| gmmaha | vhosakot: the file in /usr/local/share/kolla right? | 21:49 |
| vhosakot | yep... /usr/local/share/kolla/ansible/roles/nova/templates/nova-libvirt.json.j2 | 21:49 |
| vhosakot | gmmaha: did you stop, remove, make changes, saved file and re-deployd ? | 21:49 |
| gmmaha | vhosakot: yeah.. stop, dump containers, make cahnge to file, kolla-ansible... deploy | 21:50 |
| gmmaha | verified that the containers also got created again | 21:50 |
| * gmmaha runs to his stupid meeting | 21:50 | |
| gmmaha | vhosakot: sdake: thanks for all the help debugging | 21:50 |
| vhosakot | did it work now ? | 21:51 |
| gmmaha | have half a mind to blow out all the hosts and redeploy them | 21:51 |
| gmmaha | vhosakot: no.. but have a feeling i have to do that tonight | 21:51 |
| gmmaha | wipe the hosts and start fresh | 21:51 |
| gmmaha | :( | 21:51 |
| sdake | wait a minute | 21:51 |
| sdake | did adding -v fix the problem? | 21:51 |
| gmmaha | sdake: nope did not.. | 21:51 |
| sdake | ok good that is not an outcome i woudl hvae expected | 21:52 |
| gmmaha | sdake: :D | 21:52 |
| vhosakot | gmmaha: did nova_libvirt start fine ? | 21:52 |
| gmmaha | that would have been a bizzare one | 21:52 |
| gmmaha | vhosakot: no, it did not.. | 21:52 |
| vhosakot | sdake: -v should _not_ resolve the issue.. :) | 21:52 |
| gmmaha | stil on the reboot cycle | 21:52 |
| vhosakot | gmmaha: ah ok | 21:52 |
| sdake | so lets get some more logs this time | 21:52 |
| gmmaha | sdake: just thanking you guys for sitting with me through all this | 21:52 |
| sdake | now that we have debug enabled | 21:52 |
| vhosakot | but gmmaha is not even seeing th libvirtd.log in heka | 21:53 |
| sdake | vhosakot can you tell me where that libvirt-admin file is in your environment? | 21:53 |
| sdake | libvirtd.log wont be created becaue libvirtnever starts | 21:53 |
| sdake | becaues his envirionment is busted | 21:53 |
| vhosakot | sdake: but, he pastebinned the output of libvirtd.log from heka this morning right ? | 21:54 |
| vhosakot | gmmaha: how did you see the contents of this pastebin ? http://paste.openstack.org/show/492125/ | 21:54 |
| sdake | those wer old olgs | 21:54 |
| vhosakot | ah ok | 21:54 |
| sdake | old dogs | 21:55 |
| gmmaha | vhosakot: just dokcer logs | 21:55 |
| gmmaha | i guess | 21:55 |
| * gmmaha isnt sure what he is talking about though | 21:55 | |
| vhosakot | ok... send "docker logs nova_libvirt"... let us see if there is any debug message | 21:56 |
| manjeets | sdake: on cleaned up targets http://paste.openstack.org/show/492168/ | 22:03 |
| sdake | 25 minutes to build with aufs | 22:04 |
| sdake | rrather deploy | 22:04 |
| manjeets | its aufs already | 22:04 |
| *** achanda has joined #kolla | 22:04 | |
| manjeets | yes deploy takes around 21 to 25 mins with aufs | 22:05 |
| sdake | what about reconfigure? | 22:05 |
| sdake | time sudo ./kolla-ansible reconfigure | 22:05 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd Docker container https://review.openstack.org/298451 | 22:12 |
| *** Marga_ has quit IRC | 22:19 | |
| *** Marga_ has joined #kolla | 22:19 | |
| sdake | inc0 jpeeler rhallisey review - stablemitaka broken | 22:21 |
| sdake | https://review.openstack.org/#/c/298418/ | 22:21 |
| patchbot | sdake: patch 298418 - kolla (stable/mitaka) - Add creation of new log folder for Horizon | 22:21 |
| *** ccesario_ has joined #kolla | 22:22 | |
| inc0 | jpeeler, we ran into race condition;) | 22:23 |
| jpeeler | that's gotta be some type of gerrit bug | 22:24 |
| jpeeler | it shows now me as having +1, when I never set that | 22:24 |
| inc0 | well whatever, I hit +2 | 22:24 |
| jpeeler | er, maybe i did it by mistake | 22:24 |
| inc0 | +w | 22:24 |
| jpeeler | yeah i'm dumb, nevermind | 22:25 |
| openstackgerrit | Merged openstack/kolla: Add creation of new log folder for Horizon https://review.openstack.org/298418 | 22:27 |
| sdake | inc0 or rhallisey either of ou want to be backup release liasons for kolla | 22:30 |
| *** ccesario_ has quit IRC | 22:30 | |
| sdake | the job involves learning how the releases repoistory works and doing the job if - iI for example hae a deadly root canal infection again ;) | 22:31 |
| inc0 | yeah count me in | 22:31 |
| rhallisey | sure | 22:31 |
| sdake | well it only needs to be one person | 22:31 |
| sdake | so sort it out guys :) | 22:32 |
| inc0 | heads or tails rhallisey ?;) | 22:32 |
| rhallisey | tails | 22:32 |
| sdake | haha | 22:32 |
| vhosakot | I can help with whatever is needed... have some release/versioning/branching experience... not specific to kolla but | 22:32 |
| sdake | inc0 holds all the quarters! | 22:32 |
| rhallisey | someone filp a coin | 22:32 |
| inc0 | so if tail is oposite to side than one with president, than tails | 22:33 |
| sdake | vhosakot go find a quarter and flip it and let it lland on teh ground | 22:33 |
| inc0 | and from north dakota nonetheless! | 22:33 |
| rhallisey | ya that's tails | 22:33 |
| rhallisey | lol | 22:33 |
| sdake | and tell us the results | 22:33 |
| inc0 | one with buffalos | 22:33 |
| vhosakot | what if it lands vertically and both (neither) heads and tails | 22:33 |
| sdake | gold coins ftw | 22:33 |
| inc0 | vhosakot, then you do the job | 22:33 |
| sdake | let gravity do its thing ;-) | 22:34 |
| vhosakot | hahaha | 22:34 |
| inc0 | so tails it is, congrats Ryan;) | 22:34 |
| *** mbound has joined #kolla | 22:35 | |
| inc0 | I can be backup to your backup;) | 22:35 |
| rhallisey | lol woo :) | 22:35 |
| sdake | vhosakot was supposed to flip the coin :) | 22:35 |
| * sdake thinks inc0 just got out of oding work ;) | 22:35 | |
| inc0 | ok, waiting for him then | 22:35 |
| vhosakot | still flipping... gravity is slow today | 22:35 |
| inc0 | sdake, I could well...not volounteer if I wouldnt want it;) | 22:35 |
| sdake | inc0 i joke ;) | 22:36 |
| inc0 | sorry, my sense of humor is post-soviet | 22:36 |
| sdake | in soviet russia coin flips own you | 22:36 |
| inc0 | nah, you don't have coins at all;) | 22:37 |
| inc0 | and that is actually true in Belarus | 22:37 |
| sdake | i thin i foxtrot echo delta'ed that joke ;) | 22:37 |
| sdake | comon vhosakot i gotta do stuf f- get on twith the flipping | 22:37 |
| vhosakot | you guys serios.. | 22:38 |
| vhosakot | ok let me flip... | 22:38 |
| rhallisey | :D | 22:38 |
| vhosakot | got some quaters on my desk | 22:38 |
| inc0 | vhosakot, what made you think we're not serious? | 22:38 |
| sdake | all games are started with a coin flip ;-) | 22:38 |
| vhosakot | I first though it was a joke to flip coiin to decide release eng for kolla :) | 22:39 |
| vhosakot | ok, I'm flipping now... | 22:39 |
| vhosakot | should I start telling the rules... or assume the participants know it ? | 22:39 |
| rhallisey | he's heads im tails | 22:39 |
| *** mbound has quit IRC | 22:39 | |
| inc0 | some of them starts with spin of revolver barrel, but these usually ends with it as well | 22:39 |
| vhosakot | ok.. got it... | 22:39 |
| *** banix has joined #kolla | 22:40 | |
| vhosakot | coin is thrown once in air... in a free space... let is land on ground.. there is no carpet.. I'm on level 0... so gravity is good.... if coin stands vertically, I will redo it | 22:40 |
| rhallisey | perfect | 22:40 |
| inc0 | what's your current atmospheric pressure? | 22:41 |
| sdake | just flip a coin | 22:41 |
| vhosakot | 3 | 22:41 |
| vhosakot | 2 | 22:41 |
| vhosakot | 1 | 22:41 |
| sdake | its not rocket surgery | 22:41 |
| rhallisey | inc0, dude this is so official | 22:41 |
| vhosakot | tails | 22:41 |
| vhosakot | I got tails | 22:41 |
| rhallisey | :D | 22:41 |
| inc0 | congrats rhallisey | 22:41 |
| rhallisey | I win | 22:41 |
| rhallisey | lol | 22:41 |
| sdake | two tails | 22:41 |
| sdake | it was meant to be ;-) | 22:41 |
| rhallisey | or did I lose | 22:41 |
| vhosakot | no.. just one tails | 22:41 |
| vhosakot | I typed twice | 22:41 |
| inc0 | vhosakot, I also rollet tails | 22:41 |
| sdake | inc0 got tails eaerlier | 22:41 |
| rhallisey | XD | 22:41 |
| inc0 | rolled | 22:41 |
| inc0 | and tails from ND | 22:42 |
| sdake | what are the odds of that ;-) | 22:42 |
| rhallisey | 1/4 | 22:42 |
| inc0 | 1:4 sdake | 22:42 |
| inc0 | unless you're asking to include state of origin of a quarter, then we need to get statistics of how many coins got emitted by each state | 22:43 |
| inc0 | or at least what percentage of total coins got emitted in ND | 22:43 |
| sdake | this is where the lecture of the gambler's fallacy comes in :) | 22:43 |
| inc0 | rhallisey, you're from ND or SD? | 22:43 |
| rhallisey | what's ND/SD | 22:44 |
| inc0 | north dakota or south dakota | 22:44 |
| rhallisey | I'm from Connecticut | 22:44 |
| inc0 | lol | 22:44 |
| rhallisey | what made you think I was from there O.o | 22:44 |
| inc0 | I thought you live in one of Dakotas | 22:44 |
| inc0 | no idea | 22:44 |
| rhallisey | haha | 22:44 |
| rhallisey | that's funny :) | 22:45 |
| inc0 | then it's not nearly as cool as I thought it is...picking a quarted and rolled a tails from your home state | 22:45 |
| inc0 | I'll keep saying that you live in North Dakota because it makes this story better | 22:45 |
| rhallisey | ya that would've been cool | 22:45 |
| inc0 | sorry man | 22:45 |
| rhallisey | no problem at all | 22:45 |
| inc0 | you need to move now | 22:45 |
| rhallisey | :) | 22:46 |
| sdake | in case rhallisey and sdake have, yu know, a deadly root canal at the asm etime | 22:46 |
| rhallisey | I live in Massachusetts currently | 22:46 |
| sdake | both of you read the docs in openstack/releases | 22:46 |
| sdake | rhallisey inc0 read this review https://review.openstack.org/#/c/297976/ | 22:47 |
| patchbot | sdake: patch 297976 - releases - Add kolla releases already in Mitaka | 22:47 |
| vhosakot | haha.. gambler's fallacy... | 22:48 |
| vhosakot | rhallisey: I'm in MA as well.. :) | 22:48 |
| rhallisey | vhosakot, neat! | 22:48 |
| inc0 | there you go, we have location of our next midcycle | 22:48 |
| rhallisey | could do Red Hat westford | 22:49 |
| rhallisey | nice in the summer | 22:49 |
| sdake | or red hat ansible offices | 22:49 |
| * sdake can hook that one up | 22:49 | |
| rhallisey | oh cool | 22:49 |
| rhallisey | either way works | 22:49 |
| rhallisey | we can flip another coin | 22:49 |
| rhallisey | ! | 22:49 |
| inc0 | rh ansible would allow us to meet ansible community as well | 22:49 |
| inc0 | and express how much we....like... them | 22:50 |
| inc0 | I'll go with like | 22:50 |
| inc0 | on a recorded chat at least | 22:50 |
| vhosakot | someone in a casino once told me and my wife about gambler's fallacy.... "Just because you and your wife fought 5 times this week does not mean you guys will fight less than 5 times next week".. or something like that... it's got something to do with the frequncy of occurence of an event in a period of time | 22:53 |
| inc0 | in this particular scenerio that might actually be true | 22:54 |
| inc0 | if you fight everyday for 10 years, it may affect your fighting 2 years from then | 22:54 |
| inc0 | also name on house ownership docs | 22:55 |
| inc0 | marriage is like a handgrenade - remove a ring and house is gone | 22:55 |
| *** blahRus has quit IRC | 23:02 | |
| *** kproskurin has quit IRC | 23:07 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "Fix consistently failing CentOS gate" https://review.openstack.org/297084 | 23:07 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Test to see if this fixes the gate https://review.openstack.org/298460 | 23:07 |
| sdake | let see what the gate does with that | 23:08 |
| inc0 | sdake, so rsyslog is broken in Liberty code with docker 1.1 | 23:08 |
| inc0 | 1.10 | 23:08 |
| inc0 | we may need to backport heka | 23:08 |
| sdake | lets make 1.1.0 work - hell or highwater | 23:08 |
| inc0 | will look at heka backport | 23:09 |
| inc0 | tomorrow, when elemoine and akwasne will wake up | 23:09 |
| inc0 | reason is docker introduced /dev/shm stuff in 1.10 | 23:09 |
| inc0 | and if we bind /dev/ for rsyslog to create /dev/log | 23:09 |
| inc0 | we have issues | 23:09 |
| inc0 | I'll try other stuff | 23:10 |
| *** yuanying has joined #kolla | 23:11 | |
| sdake | inc0 we may be going about this wrong | 23:13 |
| sdake | one option which wasn't super discussed rather then trying to backport a bunch of things | 23:13 |
| inc0 | what's on your mind? | 23:13 |
| sdake | is to take mitaka as it is, and make kolla that | 23:13 |
| sdake | liberty that | 23:13 |
| sdake | and then fix the repos up and whatnot' | 23:14 |
| inc0 | well I was thinking about that | 23:14 |
| sdake | that is probably the fastest way to a working iberty tree | 23:14 |
| sdake | its not something we would ever do again | 23:14 |
| inc0 | we need to test it tho | 23:14 |
| inc0 | things may go bad, but not sure | 23:14 |
| sdake | but at this point w eare backporting a bunch of things | 23:15 |
| inc0 | let's try both ways, maybe someone can explore changing repos | 23:15 |
| sdake | i think where we need to be careful with that approach is the defaults files | 23:15 |
| *** absubram has quit IRC | 23:15 | |
| inc0 | at this point we're backporting half of mitaka anyway | 23:15 |
| sdake | its more like a backport + rewrite | 23:15 |
| sdake | why not just rewrite the damn thing | 23:15 |
| sdake | make it stable | 23:15 |
| sdake | and long term forget liberty ever happpened after operators have migrated off of it | 23:16 |
| inc0 | so best case scenerio - we just change repos and versions in our build scripts | 23:16 |
| sdake | and keep the default the same in ansible | 23:17 |
| sdake | the keystone migratoin might be a pain | 23:17 |
| inc0 | worst case scenerio - something changed so much that our code doesn't work any more | 23:17 |
| sdake | maybe we should fix the defaults file too | 23:17 |
| sdake | most of the defult changes have been around authentication | 23:18 |
| inc0 | might actually be more stable as we tested mitaka stuff much more than we will test liberty with mitaka backport | 23:18 |
| sdake | I think a careful one by one diff can sort those out | 23:18 |
| sdake | yes the testing here is critical | 23:18 |
| inc0 | sdake, I'll keep working on backport just in case, can you find someone trustworthy to try out changing repos and defaults? | 23:18 |
| sdake | because the way your goingboutit inc0 we lose allt he testing we di on liberty | 23:19 |
| sdake | and gain none that we did with mitaka | 23:19 |
| sdake | inc0 i'll take that onn as soon as i fix the gate | 23:19 |
| sdake | if it looks promising i'll look for help finishing the job | 23:19 |
| inc0 | I agree and acutally I like your plan better, I was thinking about it but I guess it got lost by "let's not backport features, only absolute minimum" | 23:19 |
| sdake | possibly you ;) | 23:19 |
| inc0 | but then backport of thin containers, upgrades | 23:19 |
| sdake | and now heka | 23:20 |
| inc0 | well, heka wasn't planned | 23:20 |
| sdake | i want backports in the future to be critical bugs only | 23:20 |
| inc0 | and I'm not positive that I can't get around it with rsyslog | 23:21 |
| sdake | but we do need to fix the state of liberty | 23:21 |
| inc0 | I just see it will be pain | 23:21 |
| sdake | liberty already has cves which we don't deploy | 23:21 |
| *** salv-orlando has quit IRC | 23:21 | |
| inc0 | we may actually look at supporting current master => lastest stable release | 23:22 |
| inc0 | so mitaka kolla will support liberty, newton will support mitaka | 23:22 |
| inc0 | most of deployment tools does that | 23:22 |
| inc0 | but that's discussion for later | 23:22 |
| sdake | you mean support n and n-1 | 23:22 |
| inc0 | or just n-1 | 23:22 |
| inc0 | we may try to support both | 23:23 |
| sdake | i want our mitaka release to implement mitaka not liberty ;-) | 23:23 |
| inc0 | ok | 23:23 |
| inc0 | well | 23:23 |
| sdake | just becaue every other deploymeent tool is behind | 23:23 |
| sdake | doesnt mean we should be | 23:23 |
| inc0 | there are good reasons for it, but again, discussion for later | 23:23 |
| inc0 | sdake, try to change repos | 23:24 |
| inc0 | I'll try to do it with ubuntu tomorrow | 23:24 |
| inc0 | hopefully any of our infra won't be "too new for liberty" | 23:24 |
| sdake | atm i am using build rom source | 23:24 |
| sdake | because ccentos is busted | 23:24 |
| inc0 | I only use build from source;) | 23:24 |
| sdake | rathe rdo is busted | 23:24 |
| inc0 | ok I'm off for today, I'm exhausted | 23:25 |
| inc0 | cya all tomorrow | 23:25 |
| *** inc0 has quit IRC | 23:26 | |
| *** shakamunyi has joined #kolla | 23:30 | |
| *** barra204 has quit IRC | 23:32 | |
| *** asalkeld_ has joined #kolla | 23:34 | |
| sdake | asalkeld you about | 23:38 |
| *** achanda has quit IRC | 23:41 | |
| *** weiyu_ has joined #kolla | 23:44 | |
| *** jtriley has joined #kolla | 23:44 | |
| *** weiyu_ has quit IRC | 23:45 | |
| manjeets | sdake question ?? | 23:46 |
| sdake | manjeets shoot | 23:46 |
| manjeets | suppose i change some params in /etc/neutron/neutron.conf | 23:46 |
| manjeets | should i just restart container ? | 23:46 |
| sdake | depends on setting o fcoy once or copy always | 23:46 |
| vhosakot | sdake: all, I am working on moving mysql logs from /var/log/ in the mariadb container to /var/log/kolla/mariadb/... can any one check if they see mysql logs in /var/log/ | 23:47 |
| sdake | if copy always you can just restart the container | 23:47 |
| vhosakot | "docker exec mariadb ls /var/log/ | grep mysql" I see myql logs in /var/log/ | 23:47 |
| manjeets | suppose i want to change some setting in conf file how do i get those reflected to actual neutron | 23:48 |
| *** weiyu_ has joined #kolla | 23:48 | |
| vhosakot | the person who commented at the end of this bug https://bugs.launchpad.net/kolla/+bug/1562231 says mysql logs are not found in /var/log/... | 23:48 |
| openstack | Launchpad bug 1562231 in kolla mitaka "lights out recovery patches may have caused problems with mariadb" [Critical,Confirmed] | 23:48 |
| manjeets | in devstack i just restart the neuton service | 23:48 |
| sdake | create config entries in /etc/kolla/config/neutron.conf | 23:48 |
| manjeets | ok | 23:48 |
| sdake | and run kolla-an sible reconfigure | 23:48 |
| *** diogogmt has joined #kolla | 23:48 | |
| manjeets | cool | 23:48 |
| vhosakot | sdake: can you send your output of "docker exec mariadb ls /var/log/ | grep mysql" | 23:49 |
| sdake | those will be merged or override tthe defaults | 23:49 |
| sdake | http://paste.fedoraproject.org/346554/59208964/ -> http://paste.fedoraproject.org/346554/59208964 | 23:49 |
| *** jtriley has quit IRC | 23:50 | |
| manjeets | sdake i assume i don't need to stop or clean-up container before i do kolla-ansible reconfigure ? | 23:50 |
| vhosakot | sdake: you don't see it..... ok.. why do i see it... http://paste.openstack.org/show/492178/ let me dig the cotnainer | 23:51 |
| vhosakot | sdake: ah, you are on CentOS right | 23:52 |
| vhosakot | they are seen in Ubuntu | 23:52 |
| sdake | vhosakot need to be seen on both equally ;-) | 23:54 |
| vhosakot | sdake: It looks like I see them on ubuntu and you dont on CentOS... http://paste.openstack.org/show/492178/ | 23:55 |
| vhosakot | yep, I will check this | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!