| *** Jeffrey4l has joined #kolla | 00:00 | |
| openstackgerrit | Martin André proposed openstack/kolla: Pin ceph-radosgw https://review.openstack.org/284475 | 00:03 |
|---|---|---|
| mandre | sbezverk: ^ this will fix your build issue | 00:03 |
| *** sdake has joined #kolla | 00:07 | |
| *** diogogmt has quit IRC | 00:10 | |
| *** blakec has joined #kolla | 00:18 | |
| *** sdake has quit IRC | 00:23 | |
| *** blakec has quit IRC | 00:46 | |
| *** kzaitsev_mb has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack/kolla: Remove information about downgrading docker https://review.openstack.org/284384 | 00:59 |
| *** iceyao has joined #kolla | 01:06 | |
| *** iceyao has quit IRC | 01:11 | |
| *** iceyao_ has joined #kolla | 01:11 | |
| *** iceyao has joined #kolla | 01:14 | |
| *** iceyao_ has quit IRC | 01:16 | |
| *** blakec has joined #kolla | 01:20 | |
| *** jasonsb has joined #kolla | 01:33 | |
| *** blakec has quit IRC | 01:34 | |
| *** blahRus has quit IRC | 01:34 | |
| *** tfukushima has joined #kolla | 01:44 | |
| *** Liuqing has joined #kolla | 01:51 | |
| *** dims has quit IRC | 01:52 | |
| *** iceyao_ has joined #kolla | 02:01 | |
| *** iceyao has quit IRC | 02:03 | |
| *** diogogmt has joined #kolla | 02:05 | |
| *** rhallisey has quit IRC | 02:22 | |
| *** iceyao has joined #kolla | 02:37 | |
| *** iceyao_ has quit IRC | 02:41 | |
| *** tfukushima has quit IRC | 02:54 | |
| *** tfukushima has joined #kolla | 03:03 | |
| *** alger has joined #kolla | 03:08 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Reconfigure keystone service https://review.openstack.org/281380 | 03:09 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add the default empty reconfigure.yml file to all roles https://review.openstack.org/283551 | 03:09 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Import module rather than funciton in the set_configs.py file https://review.openstack.org/284507 | 03:09 |
| openstackgerrit | Allen Gao proposed openstack/kolla: Move ironic-discoverd to ironic-inspector https://review.openstack.org/280655 | 03:14 |
| *** yuanying has quit IRC | 03:16 | |
| openstackgerrit | Asha proposed openstack/kolla: Fixed a typo in quickstart.rst https://review.openstack.org/284511 | 03:18 |
| *** alger has quit IRC | 03:19 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Reconfigure keystone service https://review.openstack.org/281380 | 03:21 |
| *** diogogmt has quit IRC | 03:23 | |
| *** diogogmt has joined #kolla | 03:26 | |
| *** salv-orlando has joined #kolla | 03:27 | |
| *** salv-orlando has quit IRC | 03:28 | |
| *** salv-orl_ has quit IRC | 03:30 | |
| *** alger has joined #kolla | 03:31 | |
| *** Liuqing has quit IRC | 03:37 | |
| *** Liuqing has joined #kolla | 03:38 | |
| *** diogogmt has quit IRC | 03:43 | |
| openstackgerrit | Matthew Montgomery proposed openstack/kolla: Quick spelling/grammar fix https://review.openstack.org/284522 | 03:55 |
| *** mandre has quit IRC | 03:56 | |
| *** mandre has joined #kolla | 03:56 | |
| *** dtturner has joined #kolla | 04:07 | |
| *** yuanying has joined #kolla | 04:07 | |
| *** Liuqing has quit IRC | 04:21 | |
| *** Liuqing has joined #kolla | 04:22 | |
| *** salv-orlando has joined #kolla | 04:29 | |
| *** achanda has quit IRC | 04:35 | |
| *** salv-orlando has quit IRC | 04:39 | |
| *** Marga__ has joined #kolla | 04:40 | |
| *** Marga__ has quit IRC | 04:44 | |
| *** Marga_ has quit IRC | 04:44 | |
| *** iceyao has quit IRC | 04:47 | |
| *** iceyao_ has joined #kolla | 04:47 | |
| *** alisonh has quit IRC | 04:55 | |
| *** achanda has joined #kolla | 04:55 | |
| *** Marga_ has joined #kolla | 04:56 | |
| *** Marga_ has quit IRC | 05:00 | |
| *** Marga_ has joined #kolla | 05:01 | |
| *** Marga_ has quit IRC | 05:01 | |
| *** Marga_ has joined #kolla | 05:01 | |
| *** Marga_ has quit IRC | 05:02 | |
| *** Marga_ has joined #kolla | 05:02 | |
| *** Liuqing has quit IRC | 05:08 | |
| *** Liuqing has joined #kolla | 05:08 | |
| *** alisonh has joined #kolla | 05:13 | |
| *** sdake has joined #kolla | 05:17 | |
| *** sdake has quit IRC | 05:22 | |
| *** alisonh has quit IRC | 05:32 | |
| *** alisonh has joined #kolla | 05:37 | |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add explicit calls to extend_start and skip the default https://review.openstack.org/283356 | 05:39 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Fix the running of apache based services https://review.openstack.org/283972 | 05:39 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Move nova bootstrap into nova-api https://review.openstack.org/283971 | 05:39 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: start.py: catch NoNodeError on delete https://review.openstack.org/284543 | 05:39 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add /var/log/kolla volume to koll_toolbox https://review.openstack.org/284544 | 05:39 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add missing ENV to neutron-linubridge-agent https://review.openstack.org/284545 | 05:39 |
| *** chandankumar has joined #kolla | 05:47 | |
| *** yuanying has quit IRC | 05:47 | |
| *** dims has joined #kolla | 05:52 | |
| *** salv-orlando has joined #kolla | 05:58 | |
| *** salv-orlando has quit IRC | 06:03 | |
| *** mbound has joined #kolla | 06:30 | |
| *** mbound has quit IRC | 06:34 | |
| *** dims has quit IRC | 06:43 | |
| *** achanda_ has joined #kolla | 06:45 | |
| *** dave-mccowan has quit IRC | 06:45 | |
| *** achanda has quit IRC | 06:46 | |
| *** Liuqing has quit IRC | 06:53 | |
| *** yuanying has joined #kolla | 06:54 | |
| *** salv-orlando has joined #kolla | 07:02 | |
| *** jasonsb has quit IRC | 07:02 | |
| *** salv-orlando has quit IRC | 07:07 | |
| *** alger has quit IRC | 07:11 | |
| *** salv-orlando has joined #kolla | 07:25 | |
| *** salv-orl_ has joined #kolla | 07:26 | |
| *** salv-orlando has quit IRC | 07:26 | |
| *** alger has joined #kolla | 07:28 | |
| *** tfukushima has quit IRC | 07:31 | |
| *** tfukushi_ has joined #kolla | 07:32 | |
| *** alger has quit IRC | 07:33 | |
| *** alger has joined #kolla | 07:35 | |
| *** mbound has joined #kolla | 07:43 | |
| *** daneyon_ has quit IRC | 07:55 | |
| *** jmccarthy1 has joined #kolla | 08:01 | |
| *** yuanying has quit IRC | 08:01 | |
| openstackgerrit | Lin Cai proposed openstack/kolla: Fix hard coded keystone user for nova Edit https://review.openstack.org/284575 | 08:03 |
| *** mbound has quit IRC | 08:04 | |
| *** fgimenez has joined #kolla | 08:06 | |
| *** fgimenez has joined #kolla | 08:06 | |
| *** jmccarthy1 has quit IRC | 08:08 | |
| *** yuanying has joined #kolla | 08:11 | |
| *** athomas has joined #kolla | 08:15 | |
| *** yuanying has quit IRC | 08:16 | |
| *** yuanying has joined #kolla | 08:18 | |
| *** alger has quit IRC | 08:19 | |
| *** britthouser has quit IRC | 08:21 | |
| *** jmccarthy1 has joined #kolla | 08:21 | |
| *** mikelk has joined #kolla | 08:23 | |
| *** yuanying has quit IRC | 08:26 | |
| *** yuanying has joined #kolla | 08:32 | |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: [WIP] Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 08:33 |
| *** yuanying has quit IRC | 08:38 | |
| *** yuanying has joined #kolla | 08:41 | |
| *** kproskurin has joined #kolla | 08:43 | |
| *** alger has joined #kolla | 08:46 | |
| *** yuanying has quit IRC | 08:47 | |
| *** rtmie has joined #kolla | 08:48 | |
| *** tfukushi_ has quit IRC | 08:51 | |
| ajafo | o/ | 08:56 |
| *** athomas has quit IRC | 08:59 | |
| *** achanda_ has quit IRC | 09:00 | |
| *** achanda has joined #kolla | 09:04 | |
| *** tfukushima has joined #kolla | 09:04 | |
| *** athomas has joined #kolla | 09:06 | |
| *** tfukushima has quit IRC | 09:09 | |
| *** tfukushima has joined #kolla | 09:11 | |
| *** achanda has quit IRC | 09:19 | |
| *** achanda has joined #kolla | 09:24 | |
| *** salv-orlando has joined #kolla | 09:27 | |
| *** achanda has quit IRC | 09:29 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Fix typo in dhcp-agent https://review.openstack.org/284610 | 09:30 |
| *** salv-orl_ has quit IRC | 09:30 | |
| *** pbourke has quit IRC | 09:32 | |
| *** pbourke has joined #kolla | 09:32 | |
| openstackgerrit | wangtianfa proposed openstack/kolla: A hard-coded keystone username for neutron in the neutron.conf template. Changed to use the var. https://review.openstack.org/284612 | 09:33 |
| *** alger has quit IRC | 09:35 | |
| elemoine_ | o/ | 09:35 |
| *** prithiv has joined #kolla | 09:42 | |
| *** salv-orlando has quit IRC | 09:43 | |
| *** gfidente has joined #kolla | 09:57 | |
| *** gfidente has joined #kolla | 09:57 | |
| jmccarthy1 | Morning ! | 09:57 |
| *** chandankumar has quit IRC | 09:58 | |
| *** kzaitsev_mb has joined #kolla | 10:14 | |
| *** tfukushima has quit IRC | 10:17 | |
| *** achanda has joined #kolla | 10:29 | |
| *** achanda has quit IRC | 10:35 | |
| jmccarthy1 | Question about multiple networks, I'm sure it's been discussed at some point - what is the status ? i.e. multiple neutron_bridge_names and related ? | 10:39 |
| *** bmace has quit IRC | 10:40 | |
| *** bmace has joined #kolla | 10:41 | |
| *** kzaitsev_mb has quit IRC | 10:47 | |
| *** kzaitsev_mb has joined #kolla | 10:54 | |
| *** iceyao_ has quit IRC | 10:56 | |
| *** sdake has joined #kolla | 10:58 | |
| *** salv-orlando has joined #kolla | 10:59 | |
| *** salv-orlando has quit IRC | 10:59 | |
| *** macsz has joined #kolla | 10:59 | |
| *** salv-orlando has joined #kolla | 10:59 | |
| *** slotti has joined #kolla | 11:00 | |
| *** macsz has quit IRC | 11:07 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Require nova-libvirt in nova-compute https://review.openstack.org/284642 | 11:09 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: [WIP] Customize number of instances https://review.openstack.org/284643 | 11:09 |
| *** chandankumar has joined #kolla | 11:09 | |
| *** chandankumar has quit IRC | 11:11 | |
| *** chandankumar has joined #kolla | 11:11 | |
| *** britthouser has joined #kolla | 11:13 | |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add explicit calls to extend_start and skip the default https://review.openstack.org/283356 | 11:14 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add missing ENV to neutron-linubridge-agent https://review.openstack.org/284545 | 11:14 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: start.py: catch NoNodeError on delete https://review.openstack.org/284543 | 11:14 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Fix the running of apache based services https://review.openstack.org/283972 | 11:14 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Move nova bootstrap into nova-api https://review.openstack.org/283971 | 11:14 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add ANSIBLE_NOCOLOR to chronos job https://review.openstack.org/284648 | 11:14 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Customize number of instances https://review.openstack.org/284643 | 11:21 |
| sdake | yo | 11:22 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: [WIP] Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 11:25 |
| *** sdake_ has joined #kolla | 11:29 | |
| *** sdake has quit IRC | 11:30 | |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: [WIP] Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 11:31 |
| *** sdake_ has quit IRC | 11:37 | |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Restart daemons if their config changes https://review.openstack.org/284664 | 11:50 |
| *** mdnadeem has quit IRC | 11:50 | |
| *** mdnadeem has joined #kolla | 11:51 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/kolla: Adjust package name for Ubuntu Ironic https://review.openstack.org/284672 | 12:03 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/kolla: Fix configuration files deployment in ironic-base project https://review.openstack.org/284673 | 12:03 |
| *** Liuqing has joined #kolla | 12:03 | |
| *** prithiv has quit IRC | 12:06 | |
| *** fthiagogv has quit IRC | 12:10 | |
| *** fgimenez has quit IRC | 12:16 | |
| *** Liuqing has quit IRC | 12:16 | |
| *** iceyao has joined #kolla | 12:18 | |
| *** fgimenez has joined #kolla | 12:20 | |
| *** fgimenez has quit IRC | 12:20 | |
| *** fgimenez has joined #kolla | 12:20 | |
| *** rhallisey has joined #kolla | 12:32 | |
| *** achanda has joined #kolla | 12:33 | |
| *** achanda has quit IRC | 12:40 | |
| *** prithiv has joined #kolla | 12:46 | |
| *** dwalsh has joined #kolla | 12:46 | |
| *** diogogmt has joined #kolla | 12:46 | |
| *** prithiv has quit IRC | 12:51 | |
| *** prithiv has joined #kolla | 12:53 | |
| *** fthiagogv has joined #kolla | 12:53 | |
| *** salv-orlando has quit IRC | 12:56 | |
| *** dave-mccowan has joined #kolla | 12:56 | |
| *** prithiv has quit IRC | 12:57 | |
| ccesario | Hello.... is there any document about design structure with when using kolla ?! | 12:57 |
| rhallisey | ccesario, what do you mean by design structure? | 12:58 |
| rhallisey | for a deployment? | 12:59 |
| limamauricio | hello guys, anyone else have this error when run the build on ubuntu? http://pastebin.com/e6uLuGdW | 12:59 |
| *** prithiv has joined #kolla | 12:59 | |
| ccesario | rhallisey, yes ... a deployment for a production environment .... | 13:01 |
| rhallisey | ccesario, we only really have the quickstart so far. This will be under advanced-configuration when we fill it out | 13:03 |
| rhallisey | it just hasn't been yet https://github.com/openstack/kolla/blob/master/doc/advanced-configuration.rst | 13:04 |
| sbezverk | Hello, Adding 1:ceph-radosgw-0.94.5-0.el7.centos.* into versionlock.list did not help building ceph containers is still failing, how do I activate the new version lock list? | 13:07 |
| *** diogogmt has quit IRC | 13:10 | |
| ccesario | rhallisey, ahh ok.... thanks by info.... I 'm asking because I 'm thinking in usage kolla in a new project to be developed... kolla it seems a best tool to deployment and maintenance. | 13:16 |
| ccesario | limamauricio, are you building using "binary" mode ? kolla_install_type | 13:17 |
| limamauricio | source | 13:18 |
| rhallisey | ccesario, sure I think it's a great fit. I think the best way for you to proceed is to get a feel for it and ask in channel for any advanced deployment questions you might have | 13:18 |
| rhallisey | and we will get to the docs eventually | 13:18 |
| limamauricio | source ccesario | 13:20 |
| ccesario | rhallisey, yes yes... I already did some tests using all in one deployment...... now I 'm looking for by advanced docs, as HA by example.... and the best way to usage the hardware and do not spent resources as physical deployments.... if you understand me... | 13:23 |
| rhallisey | yes | 13:24 |
| rhallisey | woops.. thought I was in terminal | 13:25 |
| rhallisey | ccesario, ya I gotcha | 13:25 |
| ccesario | limamauricio, look at... there are some comments and errors like yours https://bugs.launchpad.net/kolla/+bug/1500245/ | 13:26 |
| openstack | Launchpad bug 1500245 in kolla "Consistently Failing - Deploy OpenStack all in one node using Ansible" [Wishlist,Triaged] - Assigned to Manoj (manoj-kumar-jainman) | 13:26 |
| ccesario | rhallisey, :) | 13:27 |
| rhallisey | ccesario, SamYaple is probably you best bet for some of advice. He dev environment likely has a lot of what your looking for | 13:28 |
| ccesario | rhallisey, is there planned any kind of kolla-ui to control how and where OpenStack services are deployed across the datacenter ....!? | 13:29 |
| ccesario | rhallisey, thanks by info :) | 13:30 |
| rhallisey | ccesario, I think someone came along and was looking at it.. Right now we're trying to nail down upgrades so it hasn't been our primary focus | 13:31 |
| ccesario | nice to know.... yes yes, I read several articles about kolla where "upgrade" it was one excellent feature... | 13:35 |
| ccesario | atomic upgrades :) | 13:35 |
| *** achanda has joined #kolla | 13:37 | |
| *** achanda has quit IRC | 13:41 | |
| *** dims has joined #kolla | 13:47 | |
| openstackgerrit | Asha proposed openstack/kolla: Fixed a typo in quickstart.rst https://review.openstack.org/284511 | 13:55 |
| *** JoseMello has joined #kolla | 13:55 | |
| openstackgerrit | Asha proposed openstack/kolla: Fixed a typo in quickstart.rst https://review.openstack.org/284511 | 13:56 |
| *** salv-orlando has joined #kolla | 13:58 | |
| *** salv-orlando has quit IRC | 14:03 | |
| *** fgimenez has quit IRC | 14:07 | |
| *** slotti has quit IRC | 14:08 | |
| *** fgimenez has joined #kolla | 14:09 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Add timeout to requests.get() in kolla/cmd/build.py https://review.openstack.org/283844 | 14:10 |
| sbezverk | How do you build heka container? it seems kolla-ansible build, does not do it automatically... Now when I run deployment it fails with The requested image does not exist: 192.168.80.240:4000/kollaglue/centos-source-heka:2.0.0 | 14:11 |
| openstackgerrit | Abe Music proposed openstack/kolla: General improvements to quickstart https://review.openstack.org/284305 | 14:12 |
| elemoine_ | sbezverk: how did you build the other containers? the heka container has nothing special, it is build like any other containers | 14:17 |
| sbezverk | elemoine_ I used this command to build all other containers: kolla-build --type source --base centos --registry 192.168.80.240:4000 --push | 14:22 |
| sbezverk | but then I tried to use this: kolla-build --type source --base centos --registry 192.168.80.240:4000 --push regex heka and it created this container | 14:23 |
| elemoine_ | sbezverk: that should have built the heka container | 14:23 |
| elemoine_ | sbezverk: unless there's a bug | 14:24 |
| *** mikelk has quit IRC | 14:25 | |
| *** vhosakot has joined #kolla | 14:25 | |
| sbezverk | elemoine_ another issue during deployment, VIP is not comming up, here is logs collected from keepalive container http://paste.openstack.org/show/488188/, could you please check? | 14:28 |
| elemoine_ | sbezverk: might be related to the previous deploy error | 14:28 |
| elemoine_ | sbezverk: you can do tools/cleanup-containers to clean things up, and then redeploy the whole thing | 14:29 |
| elemoine_ | sbezverk: can you confirm that a simple kolla-build (with no regex) does not build the heka container? | 14:30 |
| sbezverk | elemoine_ the thing is I use VM with kolla to deploy all-in-one on another baremetal server, so there is no tools/cleanup-containers available on a baremetal box.. | 14:31 |
| sbezverk | elemoine_ I usually just delete everything under /var/lib/docker/* | 14:31 |
| elemoine_ | sbezverk: you can copy the cleanu-containers script on the target host and run it there | 14:32 |
| sbezverk | elemoine_ if it does not have any dependecies like ansible or something which is not part of a minimla OS image then yes.. | 14:32 |
| *** Jeffrey4l has quit IRC | 14:33 | |
| *** jtriley has joined #kolla | 14:33 | |
| elemoine_ | sbezverk: I think it's a standalone script, so copying should work | 14:33 |
| *** chandankumar has quit IRC | 14:33 | |
| *** vhosakot has quit IRC | 14:34 | |
| *** mikelk has joined #kolla | 14:35 | |
| sbezverk | elemoine_ will do that in a few minutes, but check this "Waiting for virtual IP to appear" on my deployment VM, but baremetal box has VIP up and listening: tcp 0 0 192.168.80.205:3306 0.0.0.0:* LISTEN 9640/haproxy | 14:35 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Support dependencies inside one host https://review.openstack.org/284737 | 14:35 |
| elemoine_ | sbezverk: hmm | 14:36 |
| sbezverk | elemoine_ and deployment still is waiting.. | 14:37 |
| elemoine_ | sbezverk: have you done the cleanup already? | 14:37 |
| elemoine_ | sbezverk: just for me to know… | 14:38 |
| sbezverk | elemoine_ waiting for it to fail and then will do clean up | 14:38 |
| elemoine_ | ok | 14:38 |
| *** signed8bit has joined #kolla | 14:38 | |
| elemoine_ | is kolla_internal_address correct in your globals.yml file? | 14:39 |
| elemoine_ | sbezverk: that should be your vip , and it should be available from your deploy box | 14:40 |
| elemoine_ | sbezverk: also do you use two vips? in other words, did you set kolla_external_address as well? | 14:41 |
| sbezverk | elemoine_ after clean up, the same thing.. collected proof for you: http://paste.openstack.org/show/488194/ | 14:43 |
| elemoine_ | sbezverk: I trust you | 14:43 |
| elemoine_ | sbezverk: see my other comments | 14:43 |
| sbezverk | elemoine_ I use kolla_internal_address: "192.168.80.205" and I also use: kolla_external_address: "192.168.80.25" external address is the one used to access openstack.. | 14:46 |
| *** Jeffrey4l has joined #kolla | 14:47 | |
| sbezverk | and it is bound to my management interface.. | 14:48 |
| elemoine_ | can you edit ansible/roles/haproxy/tasks/start.yml and replace kolla_internal_address by kolla_external_address in the "Waiting for virtual IP to appear" task/block? | 14:51 |
| elemoine_ | sbezverk: I am not sure that's the problem but it might be | 14:51 |
| sbezverk | elemoine_ hm it does not make sense, I mean changing it as internal address will be used to bind mariadb.. | 14:54 |
| sbezverk | elemoine_ do you know how deploy check for port up condition? | 14:54 |
| elemoine_ | sbezverk: see ansible/roles/haproxy/tasks/start.yml | 14:57 |
| elemoine_ | sbezverk: the Ansible wait_for module is used in there | 14:58 |
| *** diogogmt has joined #kolla | 14:59 | |
| elemoine_ | sbezverk: is your deploy node able to ping the internal vip (192.168.80.205)? | 14:59 |
| sbezverk | elemoine_ no I cannot ping, but I am not sure if haproxy offers this functionality.. | 15:03 |
| elemoine_ | sbezverk: and can you ping the external vip? (192.168.80.25) | 15:04 |
| elemoine_ | sbezverk: if you cannot ping the vip the wait_for task cannot do its job | 15:04 |
| sbezverk | I can ping 80.25. Well you cannot really rely on ping when you wait for a specifc TCP port?!?!? | 15:05 |
| *** chandankumar has joined #kolla | 15:05 | |
| elemoine_ | sbezverk: not what I am saying | 15:06 |
| elemoine_ | sbezverk: but if ping does not work then wait_for will probably do not work either | 15:06 |
| elemoine_ | sbezverk: did you try to change the wait_for host from internal_address to external_address as I suggested? | 15:07 |
| *** salv-orlando has joined #kolla | 15:08 | |
| *** stvnoyes has quit IRC | 15:08 | |
| *** stvnoyes has joined #kolla | 15:09 | |
| sbezverk | elemoine_ no, but it does not make sense as internal address is used to bind mariadb, anyway I will try.. | 15:09 |
| openstackgerrit | Merged openstack/kolla: Quick spelling/grammar fix https://review.openstack.org/284522 | 15:10 |
| sbezverk | elemoine_ I understand what you mean though | 15:10 |
| elemoine_ | sbezverk: I don't understand your comment | 15:10 |
| elemoine_ | sbezverk: I don't understand why it doesn't make sense to you | 15:10 |
| sbezverk | elemoine_ never mind, you need a pingable address | 15:11 |
| elemoine_ | sbezverk: this wait_for thing is just to stop the deploy process until the haproxy is available, but this step will not work if your deploy does not have access to the wait_for host | 15:12 |
| sbezverk | elemoine_ changed, still the sme thing | 15:13 |
| elemoine_ | on the target host can you confirm with netstat -nlp that there is a process bound/listening to <kolla_external_address>:<mariadb_port>? | 15:15 |
| *** salv-orlando has quit IRC | 15:15 | |
| sbezverk | netstat -tunlp | grep 3306 | 15:15 |
| sbezverk | tcp 0 0 192.168.80.205:3306 0.0.0.0:* LISTEN 23176/haproxy | 15:15 |
| elemoine_ | no entry for the external_address? | 15:16 |
| sbezverk | elemoine_ no | 15:16 |
| elemoine_ | strange isn't it? | 15:16 |
| *** inc0 has joined #kolla | 15:17 | |
| sbezverk | elemoine_ funny thing is 4 days ago exactly the same deployment wokred like a charm.. | 15:18 |
| elemoine_ | sbezverk: external_address is new, right? | 15:18 |
| sbezverk | elemoine_ the only thing changed was version of kolla, I cloned new version this morning.. | 15:19 |
| sbezverk | elemoine_ not sure I follow, what do you mean new? | 15:19 |
| elemoine_ | https://github.com/openstack/kolla/commit/34c2cb8e64c4758ec9e834dc3fabc9f5835fe9ac | 15:20 |
| elemoine_ | sbezverk: I am pretty sure this the commit that affects you | 15:21 |
| rhallisey | SamYaple, for multinode, do you set control0, 1, 2 addresses in /etc/ansible/hosts ? | 15:21 |
| openstackgerrit | Merged openstack/kolla: Fixed a typo in quickstart.rst https://review.openstack.org/284511 | 15:21 |
| inc0 | holla | 15:22 |
| *** blahRus has joined #kolla | 15:22 | |
| elemoine_ | sbezverk: so haproxy is bound to 192.168.80.205:3306 (based on netstat) | 15:23 |
| sbezverk | elemoine_ correct and this is VIP (internal) | 15:24 |
| sbezverk | elemoine_ the issue here is for 80.205 haproxy does not do arp reply.. | 15:25 |
| elemoine_ | sbezverk: ok, so I don't know right now | 15:25 |
| elemoine_ | I thought that wait_for was executed on your deploy machine, but I now think that I may be wrong | 15:26 |
| sbezverk | elemoine_ what ever outside of bare metal host will not be able to communicate until something starts sending arp replies | 15:26 |
| sbezverk | for this VIP address | 15:26 |
| rhallisey | yo yo | 15:26 |
| *** asalkeld has quit IRC | 15:27 | |
| *** asalkeld has joined #kolla | 15:27 | |
| elemoine_ | sbezverk: I'd suggest creating a kolla bug explaining your problem | 15:27 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Support dependencies inside one host https://review.openstack.org/284737 | 15:27 |
| sbezverk | elemoine_ sure, will do, thank you for your help.. | 15:27 |
| elemoine_ | sbezverk: you can also try to discuss your problem w/ someone else on the channel | 15:28 |
| elemoine_ | sbezverk: in any way i'm interested to know what the problem when you get the answer | 15:29 |
| *** Jeffrey4l has quit IRC | 15:31 | |
| *** iceyao_ has joined #kolla | 15:31 | |
| *** iceyao has quit IRC | 15:32 | |
| sbezverk | elemoine_ I will let you know for sure.. and I just filed a new bug to track this issue.. | 15:33 |
| elemoine_ | sbezverk: thanks | 15:33 |
| *** prithiv has quit IRC | 15:34 | |
| *** sdake has joined #kolla | 15:37 | |
| inc0 | SamYaple, around? | 15:37 |
| inc0 | I see strange behaviour. Outside docker networking works great, from inside container even requesting keys from apt takes minutes | 15:37 |
| inc0 | ubuntu 15.10 with ubuntu kolla | 15:37 |
| inc0 | anyone saw anything like that | 15:38 |
| inc0 | ? | 15:38 |
| *** iceyao_ has quit IRC | 15:38 | |
| inc0 | during build stage | 15:38 |
| SamYaple | inc0: yup | 15:39 |
| inc0 | what's that? | 15:39 |
| inc0 | it's horrible. | 15:39 |
| *** achanda has joined #kolla | 15:39 | |
| SamYaple | no i mean im around | 15:39 |
| inc0 | ok | 15:39 |
| inc0 | well then ^^ | 15:39 |
| SamYaple | rhallisey: i do not set hosts like that | 15:39 |
| inc0 | I just spawned 3 brand new vms | 15:39 |
| SamYaple | any slowness ive seen on that has been 1 of two things | 15:40 |
| SamYaple | slow disk, bad firewall | 15:40 |
| nihilifer | or bad storage backend for docker, but i'm not sure that's exactly that | 15:41 |
| *** sdake_ has joined #kolla | 15:41 | |
| SamYaple | nihilifer: what do yo umean "bad storage"? | 15:41 |
| inc0 | with net-host it's just fine | 15:42 |
| inc0 | can you run build over net-host? | 15:42 |
| *** sdake has quit IRC | 15:42 | |
| SamYaple | build doesnt use --net=host | 15:42 |
| inc0 | I'm running aufs, it should be good | 15:42 |
| SamYaple | it uses docker proxy | 15:42 |
| inc0 | I know, I'm asking if you can;) seems like docker proxy issue | 15:43 |
| inc0 | maybe just too slow vm | 15:43 |
| SamYaple | oh no you cant | 15:43 |
| nihilifer | SamYaple: i mean aufs - it doesn't always work well on ubuntu. but well, we can be sure now that's not causing inc0's problem | 15:43 |
| SamYaple | are you using apt-key? or like a python library? | 15:43 |
| SamYaple | nihilifer: nah inc0 is using 15.10, thats teh 4.2 kernel | 15:43 |
| SamYaple | it should be fine | 15:43 |
| openstackgerrit | Merged openstack/kolla: Pin ceph-radosgw https://review.openstack.org/284475 | 15:44 |
| inc0 | SamYaple, I'm running our build | 15:44 |
| inc0 | so apt-key | 15:44 |
| *** achanda has quit IRC | 15:45 | |
| SamYaple | inc0: `docker run --rm -it ubuntu bash` then try the apt-key | 15:45 |
| SamYaple | then llaunch one with --net=host | 15:45 |
| inc0 | doing it, same stuff | 15:46 |
| inc0 | with net host it's good | 15:46 |
| inc0 | without it's dead slow | 15:46 |
| SamYaple | i blame your iptables rules then | 15:47 |
| inc0 | top doesn't show anything strange, it doesn't seem to be CPU issue | 15:47 |
| inc0 | nothing bad there, it's clean ubuntu 15.10 | 15:47 |
| sbezverk | SamYaple should haproxy send arp replies for VIP? | 15:50 |
| inc0 | why haproxy? | 15:50 |
| openstackgerrit | Merged openstack/kolla: Adjust package name for Ubuntu Ironic https://review.openstack.org/284672 | 15:50 |
| *** jmccarthy has quit IRC | 15:51 | |
| *** jmccarthy has joined #kolla | 15:51 | |
| *** dtturner has quit IRC | 15:52 | |
| limamauricio | SamYaple, Manila does not works without /run http://pastebin.com/w7Aw5WLS | 15:52 |
| *** dtturner has joined #kolla | 15:52 | |
| SamYaple | sbezverk: haproxy nor keepalived sends arp replies | 15:55 |
| SamYaple | thats the kernel | 15:55 |
| SamYaple | limamauricio: ok, is that because of the openvswitch socket it needs to talk to? | 15:55 |
| *** daneyon has joined #kolla | 15:56 | |
| *** daneyon_ has joined #kolla | 15:57 | |
| *** sdake has joined #kolla | 15:57 | |
| *** sdake_ has quit IRC | 15:58 | |
| limamauricio | I think so, theoretically should not be necessary add the /run, but it is needed. | 15:58 |
| SamYaple | "i think so" isnt really a good answer here, can you figure out exactly what it needs please? | 15:58 |
| *** jasonsb has joined #kolla | 15:59 | |
| *** daneyon has quit IRC | 16:00 | |
| limamauricio | it needs just /run | 16:01 |
| SamYaple | why though? | 16:01 |
| sbezverk | SamYaple the reason I asked about ARP, is that I hit an issue during deploy, deploy script gets stuck waiting for VIP, but on the target host I do see VIP and port is up and in listen mode | 16:05 |
| *** prithiv has joined #kolla | 16:05 | |
| *** dims has quit IRC | 16:06 | |
| limamauricio | SamYaple, the /run is necessary for create this directory "/var/run/ openvswitch/" within manila containers | 16:07 |
| SamYaple | so keepalived will place a VIP in the form of x.x.x.x/32 on the api interface sbezverk | 16:07 |
| limamauricio | without /run I have this error "ERROR oslo_messaging.rpc.dispatcher Stderr: u'ovs-vsctl: unix: /var/run/openvswitch/db.sock: database connection failed (No such file or directory) '" | 16:07 |
| SamYaple | sbezverk you must also have another ip that has the vip in its subnet range on that same interface | 16:08 |
| SamYaple | otherwise you have to do other routing stuff | 16:08 |
| SamYaple | so, for example, vip is 192.168.10.254, you must have 192.168.10.10/24 on the same interface | 16:08 |
| *** diogogmt has quit IRC | 16:08 | |
| limamauricio | I not yet studied how openvswitch works in kolla, but the fact is that the manila needs this directory inside your container. | 16:08 |
| SamYaple | limamauricio: currently openvswitch is part of the neutron role. thats the only place it deploys | 16:09 |
| SamYaple | where is manila meant to run? | 16:09 |
| *** sdake_ has joined #kolla | 16:11 | |
| *** sdake_ has quit IRC | 16:11 | |
| *** daneyon_ has quit IRC | 16:11 | |
| *** sdake has quit IRC | 16:11 | |
| *** sdake_ has joined #kolla | 16:12 | |
| sdake_ | morning yet again | 16:12 |
| *** jmccarthy1 has quit IRC | 16:13 | |
| sbezverk | SamYaple that is exactly what I have my VIP is 192.168.80.205, my external interface is 192.168.80.25.. | 16:13 |
| *** spisarski has joined #kolla | 16:13 | |
| limamauricio | all is running localhost | 16:16 |
| limamauricio | my deployment is aio | 16:16 |
| limamauricio | manila does not needs to deploy openvswitch | 16:17 |
| SamYaple | sbezverk: if you can't ping from node1 to node2 where node2 holds the vip, you likely have a networking issue | 16:18 |
| SamYaple | it could be caused by your configuration | 16:18 |
| limamauricio | it just needs use it | 16:18 |
| SamYaple | elemoine_: regression caused by https://github.com/openstack/kolla/commit/ede53b1dcbd475aa0dff339ee8dffaa9008247a9 | 16:20 |
| SamYaple | limamauricio: using a socket means openvswitch must be deployed on the host manila is | 16:20 |
| sbezverk | SamYaple I can ping external address, what I cannot ping is VIP address, it is interesting, how you can say it is networking issue is this happens in the middle of a deployment process?? BTW here is what I see in logs of haproxy container: RRP_Instance(kolla_internal_vip) Now in FAULT state | 16:21 |
| limamauricio | Oh, so we're talking about the same thing. | 16:21 |
| limamauricio | :) | 16:21 |
| elemoine_ | SamYaple: I was away, what's the problem? | 16:22 |
| SamYaple | elemoine_: im going to fix it | 16:22 |
| SamYaple | simple mistake | 16:22 |
| SamYaple | nova_libvirt is not based on nova_base | 16:22 |
| limamauricio | SamYaple, we're talking about the same thing hahaha | 16:22 |
| elemoine_ | SamYaple: oops | 16:22 |
| SamYaple | sbezverk: i can say its a networking issue because thats waht it is. does the vip exist on a node? thats all kolla can ensure | 16:23 |
| SamYaple | sbezverk: im happy to help you out here, but more info is needed | 16:23 |
| inc0 | hmm..I found one thing that looks bad | 16:23 |
| SamYaple | lets start with `ip a` and `ip r` | 16:23 |
| elemoine_ | SamYaple: indeed, I can fix it if you want | 16:23 |
| SamYaple | elemoine_: nah testing my patch now | 16:24 |
| elemoine_ | SamYaple: sorry about that mistake | 16:24 |
| SamYaple | it didnt show in the gate because we dont use kvm in the gate elemoine_ | 16:24 |
| SamYaple | its an understandable thing to miss | 16:24 |
| limamauricio | SamYaple, sorry, all this talk was just because I did not understand your question. : / | 16:24 |
| inc0 | normal vm address is 172.16.24.58/22, docker created its network on 172.17.0.1/16...it should be exclusive but thing to consider | 16:24 |
| SamYaple | limamauricio: its ok :) | 16:24 |
| SamYaple | limamauricio: im not the best communicator over IRC apparently | 16:25 |
| SamYaple | inc0: im curious to see how you end that since those networks dont overlap... | 16:26 |
| limamauricio | I did not know that using a socket means openvswitch must be deployed on the host manila haha | 16:26 |
| SamYaple | yea openvswitch creates the socket | 16:26 |
| limamauricio | I learned something new today ^^ | 16:26 |
| inc0 | I know they don't, that's why I'm still looking | 16:26 |
| inc0 | I tcpdumped docker0 and it seems ok | 16:26 |
| inc0 | which would suggest something is doing stuff later | 16:27 |
| inc0 | brb | 16:27 |
| sbezverk | SamYaple here is requested info: http://paste.openstack.org/show/488218/ VIP address does not exist, I suspect because VRRP_Instance(kolla_internal_vip) Now in FAULT state | 16:27 |
| inc0 | or rather not doing stuff | 16:27 |
| SamYaple | sbezverk: thats a single node without a vip. i cant help you there | 16:28 |
| *** kzaitsev_mb has quit IRC | 16:28 | |
| SamYaple | sbezverk: its possible some upstream device is filtering multicast | 16:28 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: nova-libvirt is not a child of nova-base https://review.openstack.org/284793 | 16:29 |
| limamauricio | SamYaple, my last patch is ok, if it is not necessary do a rebase. The pacth is ready | 16:30 |
| limamauricio | :) | 16:30 |
| sbezverk | SamYaple this testbed was used 4 days ago for exactly the same deployment which was successful, the only change is noew version of Kolla I cloned this morning.. No, there is nothing filtering multicast.. | 16:30 |
| *** inc0 has quit IRC | 16:31 | |
| *** vhosakot has joined #kolla | 16:31 | |
| SamYaple | limamauricio: what manila agent talks to openvswitch? | 16:31 |
| *** inc0 has joined #kolla | 16:31 | |
| SamYaple | sbezverk: mind you i have done a successful deployment on mutlinode (several times) this morning. so i doubt its the kolla code | 16:32 |
| SamYaple | more likely its a configuration issue | 16:32 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/kolla: Fix /etc files location for ironic-base building from sources https://review.openstack.org/284795 | 16:32 |
| sdake_ | samyaple: [09:25:24] <SamYaple>limamauricio: im not the best communicator over IRC apparently | 16:33 |
| sdake_ | lol | 16:33 |
| SamYaple | i know exatly what im saying :/ | 16:33 |
| sdake_ | well fwiw, you are improving | 16:33 |
| *** inc0 has quit IRC | 16:33 | |
| sdake_ | keep working at it :) | 16:33 |
| sbezverk | SamYaple Is there a way to see why VRRP went into a failed state? Maybe it cold give some clues what is wrong.. | 16:33 |
| *** inc0 has joined #kolla | 16:34 | |
| SamYaple | i answer questions the way they are asked sdake_ . people should improve the asked quesstions | 16:34 |
| sdake_ | SamYaple sometimes it takes multiple questsions to get at an answer | 16:36 |
| rhallisey | lol | 16:36 |
| *** sdake_ is now known as sdake | 16:36 | |
| SamYaple | hey people can i get a quick merge on https://review.openstack.org/#/c/284793/ | 16:36 |
| sdake | you have improved your patience though when eople ask teh same question a different way :) | 16:36 |
| SamYaple | i just asnwer the same question in a different way :) | 16:37 |
| SamYaple | without new information my answer remains constant | 16:37 |
| sdake | and sometimes that helps ! | 16:37 |
| sdake | sometime it takes me 4 or 5 goes at someone to get a question answered ;) | 16:37 |
| sdake | irc is really slow form of communication | 16:37 |
| sdake | its like "fast group email | 16:38 |
| sdake | " | 16:38 |
| sdake | my job would be impossible without irc | 16:38 |
| *** salv-orlando has joined #kolla | 16:38 | |
| *** JoseMello has quit IRC | 16:38 | |
| vhosakot | Thanks for all the comments about the patch set to add timeout for requests.get() in build.py... I will add the unit test ti test the timeout! :) | 16:39 |
| vhosakot | Is there a sample ? I will refer test_build.py and the mock library | 16:39 |
| *** JoseMello has joined #kolla | 16:41 | |
| *** salv-orlando has quit IRC | 16:41 | |
| *** chandankumar has quit IRC | 16:41 | |
| *** salv-orlando has joined #kolla | 16:41 | |
| rhallisey | SamYaple, why would libvirt be the only service logging to /var/log/kolla | 16:42 |
| rhallisey | as opposing to being under nova or libvirt | 16:42 |
| SamYaple | rhallisey: its not the only one | 16:43 |
| *** absubram has joined #kolla | 16:43 | |
| sbezverk | SamYaple, is there a way to change logging level for haproxy process? | 16:43 |
| SamYaple | the only reason we have folders is for multiple logs | 16:43 |
| SamYaple | sbezverk: in the haproxy conf | 16:43 |
| SamYaple | rhallisey: libvirt only spits out libvirtd.conf | 16:43 |
| rhallisey | what other services log to /var/log/kolla . I feel like we should have dirs regardless if there are multiple services or not | 16:45 |
| sbezverk | SamYaple is it a right file? /usr/share/kolla/ansible/roles/haproxy/templates/haproxy.cfg.j2 | 16:46 |
| SamYaple | rhallisey: how does that make sense? i mean there are lots of services that log to /var/log/ directly | 16:47 |
| SamYaple | sbezverk: yea | 16:47 |
| rhallisey | SamYaple, what openstack service does that? | 16:49 |
| rhallisey | keystone mariadb log to /var/log/keystone/ and /var/log/mariadb | 16:50 |
| rhallisey | with just a single log | 16:50 |
| SamYaple | libvirt isn't an openstack service though | 16:50 |
| SamYaple | and honestly i dont like the fact its called nova_libvirt | 16:51 |
| SamYaple | this isnt the first time weve had a bug due to this | 16:51 |
| elemoine_ | SamYaple: that confused me | 16:51 |
| rhallisey | salv-orlando, legacy naming | 16:51 |
| rhallisey | SamYaple, | 16:51 |
| elemoine_ | SamYaple: the fact that it's called nova_libvirt | 16:51 |
| SamYaple | rhallisey: sure, but no reaason we have to keep it that way | 16:51 |
| SamYaple | it hurts nothing to change it | 16:51 |
| salv-orlando | rhallisey: np I'm legacy in many ways ;) | 16:51 |
| rhallisey | I'm not arguing for it, but that's why it is | 16:51 |
| SamYaple | yea i know what, i dont like it | 16:52 |
| rhallisey | salv-orlando, :D | 16:52 |
| SamYaple | lol | 16:52 |
| rhallisey | salv-orlando, legendary naming | 16:52 |
| rhallisey | there we go | 16:52 |
| *** kzaitsev_mb has joined #kolla | 16:52 | |
| limamauricio | SamYaple, manila does not have agents | 16:53 |
| limamauricio | openvswitch connects manila-share service host to neutron networks | 16:53 |
| limamauricio | to be able to talk to service VMs | 16:53 |
| *** dims has joined #kolla | 16:54 | |
| *** mikelk has quit IRC | 16:54 | |
| limamauricio | and sdake, SamYaple people should improve the asked quesstions :) | 16:56 |
| sdake | limamauricio its a two way street - a dance if you will | 16:56 |
| sdake | communication is hard - i've been at it 41 years and still struggle with younger folks ;) | 16:56 |
| limamauricio | haha | 16:57 |
| sdake | limamauricio btw, rockstar job on manilla | 16:57 |
| sdake | limamauricio can you estimate the amount of time it took yo uto come up to speed on kolla? | 16:57 |
| limamauricio | sdake, you're right, communication is hard | 16:57 |
| limamauricio | I count the time I took to do ceph work? haha | 16:59 |
| sdake | i want to understand how long it took you to become familiar with the code enough to develop with it | 16:59 |
| sdake | not the amount of time you spent getting your first deploy going | 16:59 |
| limamauricio | 15 - 20 days | 17:02 |
| limamauricio | i'm not sure | 17:02 |
| limamauricio | sdake ^ | 17:02 |
| *** fgimenez has quit IRC | 17:03 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Customize number of instances https://review.openstack.org/284643 | 17:03 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Support dependencies inside one host https://review.openstack.org/284737 | 17:03 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Dependency between nova-libvirt and nova-compute inside host https://review.openstack.org/284815 | 17:03 |
| *** fgimenez has joined #kolla | 17:03 | |
| sdake | limamauricio ok, so 80-160 hours? | 17:04 |
| sdake | at what point in the hour curve did you feel confortable wiht the code base enough to feel like you knew whatyou were doing? | 17:05 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 17:06 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Playbook for Manila upgrade https://review.openstack.org/278045 | 17:06 |
| nihilifer | ummm.. sorry for that | 17:07 |
| nihilifer | i'll restore your patchset... | 17:07 |
| limamauricio | ... | 17:08 |
| sdake | nihilifer what happened there? | 17:09 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 17:10 |
| limamauricio | nihilifer, no problem i do it :) | 17:10 |
| nihilifer | i had my patch about upgrading manila on top of that | 17:10 |
| nihilifer | and well, i just did "git review" after resolving my merge conflicts | 17:11 |
| nihilifer | without regarding limamauricio's changes | 17:11 |
| nihilifer | sorry, patchset 17 should have your newest code limamauricio | 17:11 |
| limamauricio | sdake, 120-140 hours, i felt confortable with 60-70 hours | 17:12 |
| sdake | thanks lima | 17:13 |
| sdake | so 120-140 hours is where you felt how? | 17:13 |
| sdake | comfortable at 60-70 hours I heard | 17:13 |
| sdake | but what was your feeling at the doubling of that time? | 17:13 |
| sbezverk | SamYaple could you please check this error: http://paste.openstack.org/show/488228/ | 17:13 |
| sdake | sbezverk did you do any customization with the number 80? | 17:14 |
| *** prithiv has quit IRC | 17:16 | |
| sbezverk | sdake nope, I have not done any customization :-( All I did is to removed current kolla folder, cloned a new one and run pip install kolla/ --upgrade.. | 17:16 |
| sbezverk | sdake is it possible to have conflicts if docker containers were built with older version of kolla? example 4 days ago? | 17:18 |
| *** rhallisey has quit IRC | 17:19 | |
| *** rhallisey has joined #kolla | 17:19 | |
| *** dwalsh has quit IRC | 17:20 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 17:21 |
| limamauricio | nihilifer, i update my patch :) | 17:22 |
| limamauricio | updated* | 17:22 |
| limamauricio | :) | 17:22 |
| limamauricio | now i saw that you had already restored ... my bad | 17:23 |
| *** haplo37 has joined #kolla | 17:25 | |
| sdake | sbezverk here is the feature slide in this deck: https://github.com/sdake/slides/tree/master/sdake-openstacksummit-kolla-2015-10 | 17:29 |
| sdake | just cut and paste that one if you like - its fairly accurate for mitaka as well | 17:31 |
| sdake | i will have more slides hitting that repo this morning | 17:31 |
| sdake | slide decks i mean | 17:32 |
| *** sdake_ has joined #kolla | 17:34 | |
| limamauricio | sdake, i think this bug is invalid https://bugs.launchpad.net/kolla/+bug/1544736 | 17:35 |
| openstack | Launchpad bug 1544736 in kolla "Glance: Creating the Glance service and endpoint fails" [High,Triaged] | 17:35 |
| *** achanda has joined #kolla | 17:36 | |
| *** sdake has quit IRC | 17:36 | |
| sdake_ | limamauricio I just triaged it, if its invalid add a comment | 17:37 |
| limamauricio | it use kolla_ansible | 17:38 |
| sdake_ | the state is triaged means it is assigned a priority based upon the description of theproblem, and targeted to a release | 17:38 |
| sdake_ | when it goes to confirmed, that means antoher person has confirmed it | 17:38 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Allow to override resources in global.yml https://review.openstack.org/284836 | 17:38 |
| sdake_ | bugs should only be marked invalid if it is indeed invalid - otherwise, they should be marked not enough info | 17:39 |
| limamauricio | make sense | 17:39 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Allow to override resources in globals.yml https://review.openstack.org/284836 | 17:39 |
| limamauricio | sdake, i believe it is invalid because it still use kolla_ansible, the right would use kolla_toolbox | 17:40 |
| limamauricio | sdake_ ^ | 17:41 |
| *** jasonsb has quit IRC | 17:41 | |
| sdake_ | limamauricio oh so you think it needs a retest on master? | 17:41 |
| sdake_ | limamauricio anyone can add comments to bugs, and comments are highly welcomed :) | 17:42 |
| limamauricio | sdake_, no need to test, it does not happen on the master. | 17:43 |
| limamauricio | i'll comment there | 17:43 |
| sdake_ | thanks | 17:44 |
| sdake_ | ask him to retest on masster as we think we have fixed that bug ;) | 17:44 |
| sdake_ | thats the go to I use ;) | 17:44 |
| SamYaple | sdake_: you have to stop moving al the bugs forward to the next point | 17:44 |
| SamYaple | you dont give them long enough to timeout when makes "needs more info" | 17:44 |
| SamYaple | itsl ike a 60 day timeout before they autoclose | 17:44 |
| sdake_ | I can not move the bugs that are in invalid state | 17:45 |
| sdake_ | but otherwise the bugs need to move | 17:45 |
| SamYaple | you can... | 17:45 |
| SamYaple | and you do | 17:45 |
| sdake_ | they only autoclose in the invalid state | 17:45 |
| SamYaple | you are wrong | 17:45 |
| SamYaple | the close in imcomplete | 17:46 |
| SamYaple | invalid _is_ clsoed | 17:46 |
| sdake_ | right | 17:46 |
| sdake_ | incomplete | 17:46 |
| sdake_ | ya mispoke apologies | 17:46 |
| sdake_ | i can not move the incomplete bugs to the next release | 17:46 |
| sdake_ | as in, will not | 17:46 |
| sdake_ | that way they time out | 17:46 |
| sdake_ | or are provided additional information if you like | 17:46 |
| *** sdake_ is now known as sdake | 17:46 | |
| SamYaple | you do... | 17:46 |
| SamYaple | https://bugs.launchpad.net/kolla/+bug/1500858 | 17:46 |
| openstack | Launchpad bug 1500858 in kolla "Config changes do not cause a container restart when using COPY_ALWAYS" [Wishlist,Incomplete] | 17:46 |
| sdake | yes I have been | 17:47 |
| sdake | but I can cease doing so | 17:47 |
| sdake | the rest of the bugs need to be moved forward | 17:47 |
| SamYaple | yea thats what i was asking | 17:47 |
| sdake | sure that is no problem | 17:47 |
| *** dwalsh has joined #kolla | 17:47 | |
| sdake | if bugs are in incomplete state i wont move them forward in teh future | 17:47 |
| SamYaple | the rest i get | 17:47 |
| SamYaple | though i disagree with them all being targeted somewhere | 17:47 |
| SamYaple | they should only be targeted if we are going to fix them | 17:48 |
| SamYaple | as in actively going to fix them | 17:48 |
| sdake | the downsie of that is they don't show up in the tracker list of bugs | 17:48 |
| sdake | they show up in the full bug list | 17:48 |
| SamYaple | tracker is what we are goign to finish | 17:48 |
| sdake | but they get lost | 17:48 |
| SamYaple | the fulll bug list is what we should be going by | 17:48 |
| sdake | i've moved bugs forward for oh, 2 years now ;) | 17:49 |
| limamauricio | SamYaple, manila does not have agents, openvswitch connects manila-share service host to neutron networks to be able to talk to service VMs | 17:49 |
| SamYaple | i know... | 17:49 |
| SamYaple | limamauricio: manila-share servie runs where? | 17:49 |
| sdake | if they dont get targeted, then nobody sees them and can know what to work on | 17:49 |
| SamYaple | thats not true | 17:49 |
| SamYaple | i never use that list | 17:49 |
| SamYaple | its not always valid | 17:49 |
| SamYaple | stuff falls through the crask | 17:50 |
| SamYaple | you should never trust that list | 17:50 |
| SamYaple | that list is for "we are doing these" | 17:50 |
| sdake | it wouldn't fall through the cracks if people targeted it properly | 17:50 |
| limamauricio | in the host SamTaple | 17:50 |
| limamauricio | SamYaple :) | 17:50 |
| SamYaple | sdake: thats not doing it correctly.... | 17:50 |
| SamYaple | no other group follows that practice just fyi | 17:50 |
| sdake | SamYaple we can have a further discussion about how to manage bug triage, but what we have been doing works well so far | 17:50 |
| SamYaple | are you sure anyone actually does it the way you are saying? :) | 17:51 |
| sdake | what you prpose is only confirmed bugs go into the tracker | 17:51 |
| SamYaple | ive never done it that way | 17:51 |
| SamYaple | thats not what i purpose | 17:51 |
| sdake | but nobody actually ever confirms a bug in most cases | 17:51 |
| SamYaple | they shouldnt be targeted unless they are going to be fixed | 17:51 |
| sdake | they should be targeted if they are confirmed | 17:51 |
| sdake | because they need to be fixed | 17:51 |
| sdake | prioritized from high to low | 17:52 |
| SamYaple | right, but not _that_ _cycle_ | 17:52 |
| SamYaple | not that release | 17:52 |
| sdake | people don't use the "bug" top level | 17:52 |
| SamYaple | if you are pushing bugs forward you are failing | 17:52 |
| SamYaple | all bugs in the tracker should be completed | 17:52 |
| SamYaple | or they shouldnt have been targeted | 17:52 |
| *** fgimenez has quit IRC | 17:52 | |
| SamYaple | im just telling you how launchpad projects typically work | 17:52 |
| sdake | recommend adding a session to the austin summit - we can discuss further there how to handle bug triage | 17:53 |
| SamYaple | limamauricio: i meant the manila-share container, where does it run? | 17:53 |
| SamYaple | hey rhallisey i need a quick merge on this https://review.openstack.org/#/c/284793/ | 17:54 |
| SamYaple | your comments can be addressed when we set up logging for libvirt | 17:54 |
| SamYaple | that is breaking compute for kvm | 17:54 |
| *** rmart04 has joined #kolla | 17:55 | |
| inc0 | duh...1999 Bps is not a fast networking | 17:55 |
| SamYaple | mtu issue inc0 ? | 17:56 |
| inc0 | hmm | 17:58 |
| inc0 | might be | 17:59 |
| inc0 | eth0 mtu 1450, in docker its 1500 | 17:59 |
| limamauricio | SamYale, it runs on storage node | 17:59 |
| *** kproskurin has quit IRC | 18:00 | |
| SamYaple | inc0: lol yea thats it | 18:00 |
| SamYaple | limamauricio: storage node is not going to have openvswitch running | 18:00 |
| SamYaple | it _can_ in the case of AIO, but there is no assurance it will have openvswitch | 18:00 |
| *** sdake_ has joined #kolla | 18:01 | |
| *** athomas has quit IRC | 18:02 | |
| *** sdake has quit IRC | 18:03 | |
| SamYaple | inc0: you can tcpdump for fragmented files | 18:07 |
| SamYaple | i bet youll see alot | 18:07 |
| limamauricio | SamYaple, /run create an openvswitch dir within manila_share container, it's that it needs. do the communication is a work for neutron | 18:07 |
| SamYaple | or better yet turn _off_ fragmented files and youll see no connectivity | 18:07 |
| SamYaple | limamauricio: openvswitch isnt gauranteed to run on the storage node therefore the socket may not exist | 18:08 |
| limamauricio | but only one driver needs it, generic share driver that uses Nova and Cinder | 18:09 |
| limamauricio | SamYaple ^ | 18:09 |
| limamauricio | Samyaple it connects manila-share service host to neutron networks | 18:10 |
| SamYaple | limamauricio: i suggest trying to deploy multinode where storage is its own node, youll see this doesnt work | 18:10 |
| limamauricio | in theory ;) | 18:12 |
| *** rmart04 has quit IRC | 18:12 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 18:14 |
| SamYaple | inc0: hey | 18:18 |
| SamYaple | nova upgrade | 18:18 |
| SamYaple | sighup crashes container | 18:18 |
| SamYaple | for nova-api | 18:18 |
| limamauricio | SamYaple, the role of ansible files is not guarantee that the openvswitch will be running on the node, I think it is a role of Dockerfile | 18:18 |
| SamYaple | limamauricio: youre wrong, its ansible | 18:18 |
| SamYaple | and openvswtich _shouldnt_ be on all nodes | 18:19 |
| SamYaple | jsut nodes where its needed | 18:19 |
| limamauricio | where? | 18:19 |
| limamauricio | where ansible do it | 18:19 |
| limamauricio | :/ | 18:19 |
| SamYaple | neutron role | 18:20 |
| limamauricio | ok | 18:20 |
| openstackgerrit | Rajath Agasthya proposed openstack/kolla: Recommend vagrant and heat for development only https://review.openstack.org/284856 | 18:21 |
| SamYaple | inc0: major issue with the nova upgrade | 18:22 |
| SamYaple | that sighup stuff is not working | 18:22 |
| *** rhallisey has quit IRC | 18:22 | |
| inc0 | duh | 18:22 |
| *** rhallisey has joined #kolla | 18:22 | |
| inc0 | SamYaple, you mean configs are not reloaded? | 18:23 |
| SamYaple | | 60 | nova-consoleauth | ubuntu1 | internal | enabled | up | 2016-02-25T18:22:53.000000 | - | | 18:23 |
| SamYaple | | 78 | nova-consoleauth | ubuntu3 | internal | enabled | down | 2016-02-25T18:11:12.000000 | - | | 18:23 |
| SamYaple | | 96 | nova-consoleauth | ubuntu2 | internal | enabled | down | 2016-02-25T18:11:24.000000 | - | | 18:23 |
| SamYaple | | 114 | nova-scheduler | ubuntu1 | internal | enabled | up | 2016-02-25T18:22:50.000000 | - | | 18:23 |
| SamYaple | | 117 | nova-scheduler | ubuntu2 | internal | enabled | down | 2016-02-25T18:11:29.000000 | - | | 18:23 |
| SamYaple | | 120 | nova-scheduler | ubuntu3 | internal | enabled | down | 2016-02-25T18:11:20.000000 | - | | 18:23 |
| SamYaple | i mean that, stuff isnt coming back | 18:23 |
| SamYaple | it crashed nova-api | 18:23 |
| inc0 | -.- | 18:23 |
| inc0 | hail nova | 18:23 |
| SamYaple | i think that sighup stuff is a bad idea | 18:24 |
| SamYaple | we should remove it | 18:24 |
| SamYaple | i dont think its needed anyway | 18:24 |
| inc0 | according to dansmith it is | 18:25 |
| SamYaple | no we already od a contaienr restart | 18:25 |
| SamYaple | so that a full process reload | 18:25 |
| SamYaple | i dont think its needed | 18:25 |
| sdake_ | inc0 | 18:26 |
| sdake_ | can you do me a solid | 18:26 |
| sdake_ | i know its a little annoing | 18:26 |
| sdake_ | can you add the PI for kolla-upgrade here https://review.openstack.org/#/c/283278/ | 18:26 |
| limamauricio | SamYaple, i think you still do not understand, maybe I did not explain in the best way. manila_share not needs openvswitch, only one driver needs it. | 18:26 |
| limamauricio | so, i'll do it on multinode to confirm | 18:26 |
| SamYaple | limamauricio: the logs you posted before showed manila-share trying to talk to the unix socket for openvswitch | 18:27 |
| *** kzaitsev_mb has quit IRC | 18:29 | |
| sbezverk | sdake_ I still get that error even with re-build containers. any debugging suggestions? 'AnsibleUndefinedVariable: One or more undefined variables: float object has no element 80', 'failed': True} | 18:30 |
| sdake_ | did you answer if you have 80 set somewhere in your config options in the /etc directory? | 18:31 |
| SamYaple | sbezverk: did you modify the harpxoy.cfg file? | 18:31 |
| sbezverk | sdake_ I do not have it SamYaple no I have not modified anything . | 18:32 |
| sdake_ | inc0 did you see my request above fine sir? | 18:32 |
| *** salv-orlando has quit IRC | 18:32 | |
| inc0 | sdake_, yeah, I'm on call now, will do in few minutes | 18:33 |
| sdake_ | tia | 18:33 |
| sdake_ | sbezverk reading irc log, you mentioend you were going to modify teh haproxy.cfg..j2 file | 18:34 |
| sdake_ | sbezverk I suggest double checking you didn't actuall modify it ;) | 18:35 |
| *** inc0 has quit IRC | 18:35 | |
| sbezverk | sdake_ correct, but I was just run 'cat' | 18:36 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Playbook for magnum upgrade https://review.openstack.org/275798 | 18:36 |
| sdake_ | sbezverk how did you install kolla, via the pip method? | 18:36 |
| sbezverk | sdake_ yes | 18:38 |
| sdake_ | uninstall it | 18:38 |
| sdake_ | and reinstall it | 18:38 |
| *** jtriley has quit IRC | 18:38 | |
| sdake_ | and remove /var/lib/kolla inbetween | 18:38 |
| *** salv-orlando has joined #kolla | 18:39 | |
| sdake_ | SamYaple do you think that will get him back an original haproxy file? | 18:39 |
| SamYaple | git checkout master -- ansible/roles/haproxy/templates/haproxy.cfg.j2 | 18:39 |
| sbezverk | SamYaple sdake_ done, do I need to re-build containers? | 18:43 |
| sdake_ | you dont need to rebuild containers | 18:44 |
| sdake_ | just deploy again | 18:44 |
| sdake_ | you might clean up inbetween tho ;) | 18:44 |
| sdake_ | SamYaple he is using the instructions which pip install the ansible code | 18:44 |
| sdake_ | i am not sure if kolla-ansible reads the ansible files rom /usr/share or not | 18:45 |
| sbezverk | sdake_ I ran cleanup_containers on the target box | 18:45 |
| sdake_ | good - and cleanup-host | 18:46 |
| sbezverk | sdake_ same error :-( | 18:46 |
| *** jtriley has joined #kolla | 18:46 | |
| limamauricio | SamYaple, i'll deploy a multinode environment to test it | 18:46 |
| limamauricio | :) | 18:46 |
| sdake_ | sbez do following | 18:47 |
| sdake_ | sudo updatedb | 18:47 |
| sdake_ | sudo locate haproxy.cfg.j2 | 18:47 |
| sdake_ | sbezverk ^ | 18:47 |
| limamauricio | SamYaple, you've done this deployment? | 18:47 |
| sdake_ | sbezverk then replace that file with the one in master repo | 18:47 |
| sbezverk | sdake_ I am on centos, do not see updatedb | 18:49 |
| *** jtriley has quit IRC | 18:49 | |
| *** jtriley has joined #kolla | 18:49 | |
| sdake_ | sudo yum install mlocate | 18:49 |
| limamauricio | SamYaple, openvswitch runs on compute node?? | 18:53 |
| sbezverk | sdake please see log: http://paste.openstack.org/show/488247/ haproxy in master is identical the one I am using.. | 18:53 |
| SamYaple | limamauricio: yes | 18:56 |
| limamauricio | SamYaple, oh god, you are right. i need to ensure that the storage node also do it | 18:58 |
| limamauricio | :) | 18:58 |
| *** inc0 has joined #kolla | 18:59 | |
| *** Marga_ has quit IRC | 19:00 | |
| *** daneyon has joined #kolla | 19:02 | |
| *** blahRus1 has joined #kolla | 19:02 | |
| sdake_ | sbezverk you didn't diff the /usr/share file | 19:02 |
| sdake_ | in our paste | 19:02 |
| sdake_ | our/your | 19:02 |
| inc0 | SamYaple, so reason we have sighup is to re-read update levels | 19:02 |
| inc0 | and restarting containers won't do it because by the time they restart lowest version will be still Liberty | 19:03 |
| inc0 | sighup will update it to mitaka | 19:03 |
| inc0 | so you're saying that it breaks nova services? that's crappy. | 19:03 |
| *** blahRus has quit IRC | 19:04 | |
| SamYaple | inc0: it does indeed break nova | 19:05 |
| SamYaple | nova-api completely crashes | 19:05 |
| sdake_ | sbezverk nevemrind apparently i'm blind | 19:05 |
| SamYaple | fully restarting the container would upgrade to mitaka though | 19:05 |
| sdake_ | (nearly am actually :) | 19:05 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: HAProxy and Keepalived upgrade https://review.openstack.org/283278 | 19:05 |
| SamYaple | inc0: ^^ | 19:05 |
| inc0 | we can do it as restart with serial I think | 19:05 |
| SamYaple | you would have already fixed conductor so that would be the upgrade levels | 19:06 |
| SamYaple | so if conductors are mitaka (they would be) then we would restart all other services | 19:06 |
| SamYaple | we can just drop the sighup stuff entirely | 19:06 |
| *** daneyon has quit IRC | 19:06 | |
| SamYaple | at that point all containers are upt to dat | 19:06 |
| SamYaple | date* | 19:06 |
| inc0 | SamYaple, sdake_ https://review.openstack.org/#/c/283278/ plz | 19:06 |
| dave-mccowan | question on idempotency.... for the certificate generator: is the correct behavior: if (any file that the script makes exists): bail. else: continue ? | 19:07 |
| inc0 | SamYaple, join me in #openstack-nova plz | 19:07 |
| sdake_ | inc0 i already hti +2 on it bro ;) | 19:07 |
| inc0 | thanks m8 | 19:07 |
| rhallisey | inc0, what's left to go on upgrades? | 19:08 |
| inc0 | rhallisey, if you're done with cinder | 19:09 |
| rhallisey | neutron prolly | 19:10 |
| inc0 | mariadb and ceph is on SamYaple (Sam, do you need help?) | 19:10 |
| SamYaple | inc0: ill push patches for neutron and ceph tonight | 19:10 |
| SamYaple | mariadb should mirror rabbitmq | 19:10 |
| inc0 | so find master node and tear it down last? | 19:11 |
| SamYaple | well not quite, just pick a node thats up to date and tear down last | 19:11 |
| SamYaple | difference with mariadb is you must start that node first | 19:11 |
| sbezverk | sdake_ anything else to check?? | 19:11 |
| sdake_ | sbezverk no the assumption that you modified your haproxy.cfg.j2 file is proven incorrect | 19:12 |
| sdake_ | try git diff, and see if any files are different in your repo | 19:12 |
| sdake_ | I sitll think perhaps you set something in your globals.yml which is breaking things | 19:12 |
| sdake_ | SamYaple if you see sbezverk paste above, you will see he has not modified his haproxy.cfg.j2 file | 19:13 |
| sbezverk | sdake_ here is my globals: http://paste.openstack.org/show/488253/ | 19:17 |
| sbezverk | sdake_ maybe you can spot something.. | 19:17 |
| SamYaple | inc0: looks like we can drop the sighup stuff this cycle | 19:18 |
| limamauricio | SamYaple, the configuration to run opencswitch on storage node should be on roles/neutron or roles/manila? | 19:18 |
| sdake_ | kolla_external_address: "{{ 192.168.80.25 }}" | 19:18 |
| sdake_ | why the {{ }}? | 19:18 |
| SamYaple | sdake_: that would be wrong | 19:18 |
| inc0 | but we still need restart of everything after process is done | 19:18 |
| sdake_ | sbezverk i am pretty sure that is wrong | 19:18 |
| SamYaple | it would try to parse as a variable | 19:18 |
| SamYaple | inc0: we do | 19:18 |
| SamYaple | inc0: new images | 19:18 |
| inc0 | which means after last compute node is upgraded | 19:18 |
| inc0 | no, that's not it | 19:18 |
| SamYaple | so all contaienr would be restarted | 19:18 |
| sdake_ | sbezverk kolla_external_vip_address: "{{ 192.168.80.205 }}" | 19:18 |
| *** JoseMello has quit IRC | 19:18 | |
| sdake_ | I am also pretty sure that is wrong | 19:18 |
| inc0 | we need to restart all containers AFTER we upgrade | 19:18 |
| SamYaple | inc0: we need to restart all the compute nodes again too? | 19:19 |
| inc0 | don't forget it's asyncronous | 19:19 |
| sdake_ | sbezverk out of curiosity, why did you put the {{ }} in those variables? | 19:19 |
| inc0 | yeah, that's what Dan said one day | 19:19 |
| sdake_ | is it because they were already present? | 19:19 |
| inc0 | so instead of calling sighup we can call restart, but well | 19:20 |
| sbezverk | sdake_ yes | 19:20 |
| inc0 | it's more disruptive | 19:20 |
| sdake_ | is {{ 192.168.z.z }} an ip address? :-) | 19:20 |
| sdake_ | remove the {{ }} in the variable definition and yo should be good to go | 19:20 |
| elemoine_ | what's strange is that sbezverk said it was working for him a few days ago | 19:20 |
| sdake_ | elemoine_ he repulled master it looks like | 19:21 |
| SamYaple | inc0: it will be alright | 19:21 |
| sdake_ | after the vip work hit | 19:21 |
| SamYaple | so we do all the nova-compute at the same time | 19:21 |
| SamYaple | then all the others serial, then nova-compute restart | 19:21 |
| inc0 | what do you mean by all the nova compute at same time | 19:21 |
| inc0 | ? | 19:22 |
| sbezverk | sdake_ so I remove {{}}? | 19:22 |
| elemoine_ | sdake_: Yes I know, but with this in his config file how could it work before he pulled from master... | 19:22 |
| SamYaple | i meant upgrade condcutor. restart nova-coputes serially, restart all otehr nova services serially, restart nova-compute serially | 19:22 |
| SamYaple | nova-computes appear to be the only ones that need to restart twice | 19:22 |
| SamYaple | elemoine_: kolla_external_vip_address is a new value | 19:23 |
| sdake_ | sb from those two lines i pasted above yes | 19:23 |
| SamYaple | like i said initial, mostly wrong config here | 19:23 |
| elemoine_ | SamYaple: oh right, I thought we were talking about kolla_external_address here | 19:24 |
| inc0 | SamYaple, no, all of them needs 2 restarts | 19:25 |
| inc0 | we upgrade conductors as first step | 19:25 |
| inc0 | then we upgrade control plane | 19:25 |
| inc0 | then we upgrade compute | 19:26 |
| inc0 | then we reload configs when all of it is up-to-date | 19:26 |
| inc0 | reload configs on every service | 19:26 |
| inc0 | which means we can restart in serial | 19:26 |
| SamYaple | im saying upgrade conductors then compute | 19:26 |
| SamYaple | the the control plane stuff neds single restart | 19:26 |
| sdake_ | inc0 that approach sounds correct to me | 19:27 |
| inc0 | I mean it needs restart during container redeployment and forced restart | 19:27 |
| sdake_ | SamYaple could you expand on why control palne needs single restart | 19:27 |
| inc0 | so 2 restarts total | 19:27 |
| SamYaple | inc0: right, thats my point, lets skip the 2 restarts. single restart only | 19:27 |
| SamYaple | upgrade conductors, upgrade compute, upgrade control plan restart contductors and compute | 19:28 |
| SamYaple | no restart needed on control plane since computes will have been upgraded | 19:28 |
| inc0 | not really | 19:28 |
| SamYaple | 19:21:07 < SamYaple> dansmith: to clear up some confusion. we would need to restart/sighup all nova services after the last nova-compute node is upgraded. does that mean _also_ the nova-compute services again? | 19:28 |
| inc0 | because upgrade will always be async | 19:28 |
| SamYaple | he said yes | 19:28 |
| SamYaple | it just means we need ot upgrade nova-compute after conductors | 19:29 |
| SamYaple | then the other services will restart at the newest level | 19:29 |
| *** salv-orlando has quit IRC | 19:29 | |
| sbezverk | sdake_ after removing {} It went further but got stuck again at the waiting for VIP place.. | 19:29 |
| inc0 | hold on | 19:30 |
| inc0 | if we have 2 APIs | 19:30 |
| sdake_ | sbezverk you need to provide more data then that | 19:30 |
| sdake_ | how about a paste of where its sstuck | 19:30 |
| sbezverk | sdake_ so I am back to the original issue from this morning :-( | 19:30 |
| inc0 | and restart one then restart other because ansible doesn't sync this stuff | 19:30 |
| sdake_ | i dont know what original issue you had | 19:30 |
| sdake_ | but the {{ }} is wrong | 19:30 |
| inc0 | one we restart first will still consider itself to be Liberty | 19:30 |
| inc0 | because other one was Liberty on the moment it restarts | 19:30 |
| SamYaple | inc0: oh i see the confustion here | 19:30 |
| SamYaple | yea we need multiple steps for the serial launching | 19:31 |
| SamYaple | not just one big serial playbook like you have now | 19:31 |
| sbezverk | sdake_ it would probably a good idea to get rid of {{}} in the original files, it is a bit confusing whther you need them or not.. | 19:31 |
| inc0 | so we need to ensure everything is up to date and then restart again | 19:31 |
| limamauricio | SamYaple, to deploy openvswitch in storage node add inventory_hostname in groups['storage'] in neutron roles files would be correct? | 19:31 |
| inc0 | it is serial now | 19:31 |
| SamYaple | we only need to ensure conductors and compute are up to date inc0 | 19:31 |
| inc0 | what we can't do is to have reload as single step | 19:31 |
| SamYaple | then upgrade/restart the api/scheduler stuff | 19:32 |
| SamYaple | then restart the conductors and compute again | 19:32 |
| SamYaple | so api/scheduler only restarts ones | 19:32 |
| SamYaple | the conductor and compute restart twice | 19:32 |
| inc0 | so I'm not sure how upgrade_levels are implemented | 19:32 |
| inc0 | but if scheduler controls version of coputes as well | 19:33 |
| inc0 | we need 2 restarts | 19:33 |
| sbezverk | sdake_ when I start deploy it get stuck on [haproxy | Waiting for virtual IP to appear] and then script fails after time out gets expired.. but on the target system I do see VIP address and port in listen.. | 19:33 |
| inc0 | if it does not, just one | 19:33 |
| SamYaple | inc0: ok sounds like you have to question to ask | 19:33 |
| sdake_ | sbezverk paste your globals.ymla gain plz | 19:33 |
| SamYaple | i agree with your assesment | 19:33 |
| sbezverk | sdake_ here you go: http://paste.openstack.org/show/488256/ | 19:35 |
| *** vhosakot has quit IRC | 19:35 | |
| sdake_ | sbezverk I am pretty sure but not positivey our external and internal VIPs need to be separate IPs | 19:35 |
| sdake_ | what is located at ip address .205? | 19:35 |
| inc0 | SamYaple, no, now when I think about it, we need 2 restarts whatsoever | 19:35 |
| inc0 | however* | 19:35 |
| sdake_ | the target node or nothing? | 19:35 |
| inc0 | because of this async thing I mentioned before | 19:35 |
| limamauricio | openvswitch is deployed on compute node through inventory_hostname in groups['compute'] in neutron roles files? | 19:36 |
| inc0 | if you have 2 apis, there is no guarantee that container upgrade will trigger correct upgrade_version detection | 19:36 |
| *** Marga_ has joined #kolla | 19:36 | |
| inc0 | even if all APIs care about are APIs | 19:36 |
| *** salv-orlando has joined #kolla | 19:36 | |
| inc0 | because upgrade between nodes might be async | 19:36 |
| inc0 | so we need 2 restarts, and since we do, we might do it after all is done | 19:37 |
| *** vhosakot has joined #kolla | 19:37 | |
| sbezverk | sdake_ 80.205 is VIP | 19:39 |
| sdake_ | so an unused IP address on your network? | 19:39 |
| sdake_ | you need a .206 VIP as well for external vip address | 19:39 |
| sbezverk | sdake_ I collected logs from haproxy and keepalive containers http://paste.openstack.org/show/488257/ | 19:39 |
| sdake_ | sbezverk thanks for doing that, but until you have separate ips for kolla_internal_address and kolla_external_vip_address it doesn't help :) | 19:40 |
| sdake_ | what is 192.168.80.25? | 19:40 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: [WIP] Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 19:41 |
| inc0 | SamYaple, let's do this, just to be on the safe side: refactor reload.yml to reload controllers and reload compute | 19:41 |
| *** pbourke- has joined #kolla | 19:41 | |
| sbezverk | it is ip on my management interface and it is used to access opensatck? | 19:41 |
| inc0 | and reload compute first with serial 10% and then reload controllers with serial 30% | 19:41 |
| inc0 | reload will restart each container, no upgrade, just docker restart | 19:41 |
| sbezverk | sdake_ let me try to add 80.206 | 19:41 |
| sdake_ | you can't just blindly add 206 | 19:42 |
| sdake_ | you hav eto make sure 206 is not an address in use | 19:42 |
| sdake_ | sbezverk are you deploing from one node to another? | 19:42 |
| sbezverk | sdake_ That is correct, I use deployment VM and pushing containers to a baremetal server. | 19:44 |
| sbezverk | sdake_ new globals.yml, same result http://paste.openstack.org/show/488259/ | 19:45 |
| sdake_ | sbez what is on .25? | 19:45 |
| sdake_ | is that your baremetal server? | 19:46 |
| sbezverk | sdake_ .25 yes | 19:46 |
| sdake_ | kolla_external_address should be .206 | 19:46 |
| sdake_ | or .205 | 19:47 |
| sdake_ | but definately not .25 | 19:47 |
| sdake_ | the docs seem to indicate it should be .205 | 19:47 |
| sdake_ | but I think it should be .20 | 19:47 |
| sdake_ | .206 | 19:47 |
| sdake_ | dave-mccowan any thoughts ^^ | 19:47 |
| SamYaple | inc0: thats the quickest solution. we will optimize later | 19:49 |
| sbezverk | sdake_ 80.25 is bound to management interface, which is supposed to be used to access horizon on baremetal server, I thought 80.205 will be used for internal opensatck services.. not understand why would I need and 80.206, but if it help to make things working, then why not.. | 19:49 |
| inc0 | optimize would mean we need to figure out sighup | 19:49 |
| inc0 | that's the end solutio | 19:49 |
| inc0 | n | 19:49 |
| SamYaple | inc0: eh whatever. lets just get it working | 19:49 |
| sdake_ | sbezverk the idea is you ahve an internal management network, and an external API network | 19:50 |
| inc0 | yeah, that's my point | 19:50 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 19:50 |
| sdake_ | dave-mccowan can probably do a better job of explaining then I can | 19:51 |
| sdake_ | or the docs explain it | 19:51 |
| sdake_ | but the external_address should not be .25 | 19:51 |
| *** ayoung has quit IRC | 19:51 | |
| sbezverk | sdake_ in all_in_one scenario everything should be able to collapse to three interface 1 - management including internal openstack services, 2- neutron external/provider interface 3 - tenants | 19:51 |
| sdake_ | inc0 re the 20 node setup | 19:51 |
| SamYaple | sbezverk: please post your globals.yml, inventory, and `ip a` and `ip r` from each node | 19:51 |
| sdake_ | inc0 did you haea chance to get kolla deployed on it | 19:51 |
| dave-mccowan | unless you have a reason to use two VIPs, you're probably better off not using two VIPs | 19:52 |
| SamYaple | sbezverk: AIO can be single interface, or easily two interfaces | 19:52 |
| sdake_ | there is no tenant interface | 19:52 |
| inc0 | just get access to it, I'll deploy 1.1.0 | 19:52 |
| SamYaple | sdake_: tehre is a tenant interface | 19:52 |
| sdake_ | i know | 19:52 |
| sdake_ | but he doesn't need to muck with that | 19:52 |
| sdake_ | sbezverk look at the original configuration file | 19:53 |
| inc0 | and then, after Mitaka release I'll upgrade it to mitaka | 19:53 |
| sdake_ | any thing with interface in it that wa commented out, recommend out :) | 19:53 |
| sdake_ | recomment out :) | 19:53 |
| inc0 | but I will have 20 nodes for our talk | 19:53 |
| dave-mccowan | unless you're using NAT or DNS, leave out kolla_external_address, and all.yaml will take care it for you. | 19:53 |
| sdake_ | inc0 i'm just curious what the time is to deploy on 20 nodes | 19:53 |
| inc0 | will let you know | 19:53 |
| sdake_ | dave-mccowan and klla_external_vip_address correct? | 19:54 |
| sbezverk | SamYaple Here is requested info: http://paste.openstack.org/show/488261/ | 19:54 |
| sdake_ | sbezverk kolla will choose smart defaults for anything in the globals.yml that is commented out | 19:54 |
| sdake_ | sbezverk anything that isnt' commented out must be set | 19:54 |
| sdake_ | that should probably go at the top of globals.yml ;_) | 19:54 |
| SamYaple | sbezverk: external_adress should be dns name that maps to 192.168.80.206 | 19:55 |
| dave-mccowan | unless you want two vips for a reason (like security in production), comment out kolla_external_vip_address and defaults will do the right thing. | 19:55 |
| dave-mccowan | if you want two vips, set kolla_internal_address and kolla_external_vip_address to different IPs that are not in use anywhere else. set network_interface and external_vip_interface appropriately. | 19:56 |
| SamYaple | sbezverk: kolla_external_address conflicts with address on mgmt_br0 | 19:56 |
| sdake_ | sbezverk back to basics here, don't use the second VIP | 19:57 |
| sbezverk | SamYaple ok if I set kolla_external_address conflicts to 192.168.80.205 and do not use any reference to 80.25, the issue should be resolved right? | 19:57 |
| SamYaple | sbezverk: i mean yea i suppose so. i would just recommend commenting out the kolla_external_address and kolla_external_vip_address variables | 19:58 |
| SamYaple | the defaults are what you want | 19:58 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 19:58 |
| SamYaple | limamauricio: can you point me to some docs as to what manila-share needs to interact with openvswitch for? | 19:59 |
| sdake_ | sbezverk take sam's recommendation of commenting out kolla_external_address and kolla_external_vip-address, sdake and dave-mccowan agree | 19:59 |
| sdake_ | sbezverk dont forget to cleanup your containers ;-) | 20:00 |
| limamauricio | SamYaple, I thought I'd add a condition to implement "ovs" in the storage node only when the enable_manila = yes | 20:02 |
| SamYaple | limamauricio: and if you just run with manila, it wont deploy openvswitch | 20:03 |
| SamYaple | no what needs to happen is openvswitch needs to be its own role if more than just neutron needs it | 20:03 |
| SamYaple | however i would like some docs as to its interaction | 20:03 |
| sbezverk | SamYaple with new kolla_internal_address 192.168.80.205, I see the same issue. http://paste.openstack.org/show/488265/ | 20:04 |
| sbezverk | SamYaple but VIP address and ports show as LIsten on target host: see this: http://paste.openstack.org/show/488267/ | 20:06 |
| limamauricio | SamYaple https://review.openstack.org/#/c/273724/ | 20:08 |
| limamauricio | if you see it better can use it http://52.34.178.242:8000/html/manila-share-install.html | 20:08 |
| SamYaple | limamauricio: that says the neutron agent must be running on the same node as manila-share | 20:10 |
| limamauricio | yes | 20:10 |
| SamYaple | thats a whole different situation | 20:10 |
| limamauricio | but manila-share should run in storage-node | 20:10 |
| SamYaple | i frankly dont have time to look into this but this make neutron a required role to run before manila | 20:11 |
| limamauricio | I tried to explain this to you, but I could not. sorry :/ | 20:11 |
| dave-mccowan | sdake_ question on idempotency.... for the certificate generator: is the correct behavior: if (any file that the script makes exists): bail. else: continue ? | 20:13 |
| SamYaple | dave-mccowan: not quite | 20:13 |
| SamYaple | its not any file it makes, its what does that particular action make | 20:13 |
| SamYaple | so if you break after creating file1, but the end result is file1 file2 file3, running again should generate file2 and file3 | 20:14 |
| limamauricio | neutron agent is required to generic driver manage a share server | 20:14 |
| dave-mccowan | SamYaple. i'm worried about ending up with mismatched certificates... where the CA.crt doesn't match the Server.pem. | 20:14 |
| sdake_ | afk helping on sbezverk in a screen share dave-mccowan | 20:15 |
| SamYaple | dave-mccowan: youd need to write checks for that then | 20:15 |
| sdake_ | my take is sort of - but I'll have to get back to you | 20:15 |
| SamYaple | alright guys. i have a bit of an issue :/ | 20:15 |
| *** Marga_ has quit IRC | 20:15 | |
| SamYaple | i think we need to update kolla_internal_address to kolla_internal_vip_address | 20:15 |
| *** Marga_ has joined #kolla | 20:15 | |
| SamYaple | and make kolla_intenral_address a dns name | 20:15 |
| SamYaple | or we need to rename kolla_external_vip_address | 20:15 |
| SamYaple | its just to confusing | 20:16 |
| *** inc0 has quit IRC | 20:16 | |
| dave-mccowan | SamYaple I agree. trade off between backward compatible and confusing. maybe we can make the current names aliases that we deprecate? | 20:17 |
| SamYaple | i think we need to keep kolla_internal_address the same | 20:17 |
| dave-mccowan | i like kolla_internal_vip_address and kolla_external_vip_adress. | 20:17 |
| limamauricio | SamYaple, I thought I'd add a condition to implement "ovs" in the storage node only when the enable_manila = yes, it will resolve | 20:17 |
| SamYaple | yea but least impactful change is changing the other ones | 20:18 |
| *** pbourke- has quit IRC | 20:18 | |
| SamYaple | dave-mccowan: im not sure yet | 20:18 |
| SamYaple | but we need to do something | 20:18 |
| limamauricio | just deploy ovs on storage node if enable_manila = yes SamYaple | 20:18 |
| SamYaple | limamauricio: its not just ovs, its neutron agent and ovs | 20:18 |
| *** salv-orlando has quit IRC | 20:18 | |
| limamauricio | yes yes | 20:18 |
| limamauricio | both | 20:18 |
| limamauricio | so you think you add inventory_hostname in groups [ 'storage'] and check condition is not the right way? SamYaple | 20:20 |
| *** salv-orlando has joined #kolla | 20:20 | |
| dave-mccowan | SamYaple part of the solution will be a section of document with examples of the different possibilities | 20:20 |
| SamYaple | dave-mccowan: what about kolla_internal_dns_name and kolla_external_dns_name | 20:24 |
| SamYaple | kolla_internal_address and kolla_extnerla_address | 20:24 |
| SamYaple | where the later two are teh vip | 20:24 |
| *** Marga_ has quit IRC | 20:25 | |
| dave-mccowan | do you think it's common to use a NAT address as kolla_external_address today? or is steve the only one who does that? | 20:25 |
| *** kzaitsev_mb has joined #kolla | 20:26 | |
| *** Marga_ has joined #kolla | 20:26 | |
| SamYaple | actually i think i have it for backwards compatibility | 20:26 |
| SamYaple | let me submit a patch | 20:26 |
| *** Marga_ has quit IRC | 20:27 | |
| *** gfidente has quit IRC | 20:39 | |
| *** daneyon has joined #kolla | 20:40 | |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 20:42 |
| *** JoseMello has joined #kolla | 20:42 | |
| *** haplo37 has quit IRC | 20:43 | |
| openstackgerrit | Merged openstack/kolla: Fix /etc files location for ironic-base building from sources https://review.openstack.org/284795 | 20:52 |
| *** Marga_ has joined #kolla | 20:55 | |
| *** Marga_ has quit IRC | 20:59 | |
| *** pbourke- has joined #kolla | 21:02 | |
| *** haplo37 has joined #kolla | 21:06 | |
| sdake_ | dave-mccowan pretty sure i'm the only one who does that ;-) | 21:12 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka send logs to Elasticsearch https://review.openstack.org/284188 | 21:14 |
| daneyon | SamYaple are you there? | 21:16 |
| *** pbourke- has quit IRC | 21:20 | |
| sdake_ | sbezverk I had another thought, perhaps your network device is bokred in some way | 21:21 |
| sdake_ | a recycle could do the trick | 21:21 |
| sdake_ | sbezverk another option is to use the address .207 | 21:21 |
| sdake_ | since the arp tables wouldn't have been corruptd by 205/206 | 21:21 |
| *** Marga_ has joined #kolla | 21:22 | |
| *** achanda has quit IRC | 21:23 | |
| SamYaple | daneyon: here i am! | 21:25 |
| daneyon | SamYaple thx | 21:25 |
| *** pbourke- has joined #kolla | 21:25 | |
| sdake_ | yo pbourke | 21:25 |
| pbourke- | sdake_: hey! | 21:25 |
| daneyon | i'm presenting kolla to a meetup group tonight... scary :-o | 21:25 |
| SamYaple | daneyon: oh that is scary | 21:25 |
| SamYaple | does it work? | 21:25 |
| pbourke- | course it does | 21:26 |
| daneyon | SamYaple do you have a kolla demo machine or a recorded demo of kolla? My recording is super ld and the one you and sdake did in Tokyo is a bit much for my audience | 21:26 |
| SamYaple | daneyon: I do not have a recording or a remotely accessable demo machine at this time | 21:27 |
| *** Marga_ has quit IRC | 21:27 | |
| daneyon | SamYaple OK | 21:27 |
| SamYaple | one of the other folks here might | 21:27 |
| daneyon | SamYaple sdake does this doc still work: http://docs.openstack.org/developer/kolla/heat-dev-env.html | 21:29 |
| *** achanda has joined #kolla | 21:31 | |
| *** achanda has quit IRC | 21:33 | |
| SamYaple | daneyon: i dont think so. but wait for sdake to confirm. i dont think anyones used it in a long time | 21:34 |
| SamYaple | i never have successfully used it | 21:35 |
| sdake_ | SamYaple does this change make any sense to you https://review.openstack.org/#/c/284836/2 | 21:35 |
| *** sdake_ is now known as sdake | 21:35 | |
| sbezverk | sdake_ if going to the older version does not help, I will start moving physical things around.. I am still rebuilding containers.. | 21:35 |
| SamYaple | sdake: you know thats mesos right? | 21:37 |
| sdake | SamYaple yes for some reason I thought it was in kolla repo | 21:37 |
| *** achanda has joined #kolla | 21:37 | |
| *** pbourke- has quit IRC | 21:37 | |
| SamYaple | sdake: can you respond to daneyon ? | 21:38 |
| sdake | SamYaple i already did | 21:38 |
| sdake | he followed that up by asking you | 21:38 |
| SamYaple | hmm i dont see that in my logs ok. | 21:39 |
| daneyon | sdake my latest question is related to http://docs.openstack.org/developer/kolla/heat-dev-env.html | 21:39 |
| SamYaple | right | 21:39 |
| daneyon | sdake is the above doc accurate? | 21:39 |
| sdake | daneyon i see, well heat and vagrant, I'd say are bust | 21:39 |
| sdake | daneyon so I doubt that heat works at all, and YMMV with vagrant | 21:40 |
| daneyon | sdake ok | 21:40 |
| sdake | nobody maintains it | 21:40 |
| *** limamauricio_ has joined #kolla | 21:43 | |
| *** opennode has joined #kolla | 21:45 | |
| *** Marga_ has joined #kolla | 21:51 | |
| openstackgerrit | Merged openstack/kolla: General improvements to quickstart https://review.openstack.org/284305 | 21:52 |
| openstackgerrit | Merged openstack/kolla: Import module rather than funciton in the set_configs.py file https://review.openstack.org/284507 | 21:53 |
| daneyon | sdake SamYaple is docker hub still the place where the official kolla images live? It looks like it, but just want to double-check. | 21:55 |
| sdake | daneyon we don't push official images at this time | 21:55 |
| sdake | daneyon i am working to get it done in the gate | 21:56 |
| sdake | there are crednetial issues which need to be sorted out | 21:56 |
| *** rhallisey has quit IRC | 21:57 | |
| daneyon | sdake ok... so no official repo to pull, users need to build their own from src. | 21:58 |
| sdake | or from binary | 21:58 |
| sdake | either works | 21:58 |
| sdake | more or less | 21:58 |
| limamauricio_ | SamYaple, i'm thinking revert my patch and create a new one to solve the problem of Manila, neutron agent and openvswitch | 21:58 |
| openstackgerrit | Merged openstack/kolla: Add the default empty reconfigure.yml file to all roles https://review.openstack.org/283551 | 21:58 |
| limamauricio_ | What do you think about this? SamYaple | 21:59 |
| *** dwalsh has quit IRC | 22:00 | |
| *** macsz has joined #kolla | 22:09 | |
| *** achanda has quit IRC | 22:09 | |
| openstackgerrit | Yathiraj Udupi proposed openstack/kolla: Added a Nova guide https://review.openstack.org/284961 | 22:18 |
| openstackgerrit | Yathiraj Udupi proposed openstack/kolla: Added a Nova guide https://review.openstack.org/284961 | 22:22 |
| *** opennode has quit IRC | 22:22 | |
| *** vhosakot has quit IRC | 22:23 | |
| *** vhosakot has joined #kolla | 22:31 | |
| *** macsz has quit IRC | 22:32 | |
| openstackgerrit | Richa proposed openstack/kolla: Fixed grammatical error in ceph-guide.rst https://review.openstack.org/284968 | 22:32 |
| *** jtriley has quit IRC | 22:32 | |
| sdake | sbezverk found a legimiate keepalived bug | 22:34 |
| sdake | git blame shows keepalived was changed feb 16 to use socat | 22:34 |
| *** rhallisey has joined #kolla | 22:35 | |
| sbezverk | sdake Here is the bug I did I filed this morning.. https://bugs.launchpad.net/kolla/+bug/1549857 | 22:37 |
| openstack | Launchpad bug 1549857 in kolla "deployment waiting for VIP address until timed out" [Undecided,New] | 22:37 |
| openstackgerrit | Richa proposed openstack/kolla: Fixed grammatical error in ceph-guide.rst https://review.openstack.org/284968 | 22:37 |
| sdake | sbezverk are you going t o fix that bug or wnat someone else to take care of it? | 22:42 |
| sbezverk | sdake if you do not mind and have a bit of time to walk me through the process, I would like to fix it, if it is urgent then I guess somebody else should do it | 22:43 |
| sdake | i can work with you friday to get setup | 22:44 |
| sdake | if your available then | 22:44 |
| sdake | int he afternoon | 22:44 |
| sdake | you found the bug and found the fix, ou should get credit f or it ;) | 22:44 |
| sdake | i would have never thought of running keepalived separately from a container | 22:45 |
| sdake | never is a long time, i mean I may have got there eventually | 22:45 |
| sbezverk | sdake sure I have time tomorrow or tonight | 22:45 |
| sbezverk | it is up to you | 22:46 |
| *** absubram has quit IRC | 22:48 | |
| sdake | tonight then | 22:48 |
| sdake | i usually work quite a bit | 22:49 |
| sdake | so i'll be around | 22:49 |
| openstackgerrit | Richa proposed openstack/kolla: Fixed grammatical error in ceph-guide.rst https://review.openstack.org/284968 | 22:49 |
| sdake | I find it fascinating how much our docs improve in the last 2-3 weeks prior to a cycle ending | 22:52 |
| sdake | SamYaple ^^ | 22:52 |
| openstackgerrit | Richa proposed openstack/kolla: Fixed grammatical error in ceph-guide.rst https://review.openstack.org/284968 | 22:53 |
| openstackgerrit | Richa proposed openstack/kolla: Fixed grammatical error in ceph-guide.rst https://review.openstack.org/284968 | 22:54 |
| vhosakot | doc merge blast :) | 22:54 |
| *** ayoung has joined #kolla | 23:03 | |
| *** achanda has joined #kolla | 23:04 | |
| openstackgerrit | Merged openstack/kolla: HAProxy and Keepalived upgrade https://review.openstack.org/283278 | 23:05 |
| SamYaple | sdake: feb 16.... 2015 it was changed to use socat | 23:07 |
| SamYaple | its been like that for a long fine | 23:07 |
| SamYaple | i mean its possible centos _never_ worked... i guess? | 23:08 |
| *** salv-orl_ has joined #kolla | 23:08 | |
| sdake | perhaps socat isnt' it | 23:09 |
| sdake | i was reading 2016 | 23:09 |
| sdake | not 2015 ;) | 23:09 |
| SamYaple | yea socat has been there since inc0 initially implemented keepalived | 23:09 |
| SamYaple | i known because i fought him on it | 23:09 |
| SamYaple | nothing has changed recently at all | 23:10 |
| SamYaple | socat exists in teh centos container | 23:10 |
| SamYaple | i really dont know what you think youve found | 23:10 |
| sdake | possible i am wrong | 23:11 |
| sdake | we are looking into it, will let you know | 23:11 |
| sdake | the ^^ was for the doc comment ;) | 23:11 |
| *** blahRus1 has quit IRC | 23:11 | |
| SamYaple | yea im jsut letting you know. this is likely user error.... | 23:11 |
| *** salv-orlando has quit IRC | 23:12 | |
| sdake | ya socat is in the keepalived container not the base container | 23:13 |
| sdake | the odd thing is, he took the keepalived config and runs it baremetal, that works | 23:13 |
| sdake | he runs in container, that doesn't work | 23:13 |
| sdake | not sure how thats user error ;) | 23:14 |
| SamYaple | selinux likes to break that container | 23:14 |
| SamYaple | you sure thats not turned on? | 23:14 |
| sdake | selinux permissive | 23:14 |
| SamYaple | and mind you, it could be an outdated container | 23:14 |
| SamYaple | when was the last rebuild? | 23:14 |
| openstackgerrit | Dave McCowan proposed openstack/kolla: Add Ansible scripts to generate TLS certificates for testing https://review.openstack.org/282733 | 23:14 |
| sdake | 37 minutes ago | 23:14 |
| SamYaple | I am doubtful this is a problem with Kolla, or rather one Kolla can fix with a change to its repo | 23:15 |
| dave-mccowan | sdake ^^^ idempotency for the win | 23:15 |
| sdake | has something to do with the check_alive script | 23:16 |
| SamYaple | sure. its probably looking in the wrong location/wrong volume | 23:17 |
| SamYaple | only thing thats chaged there is named volume usage for the socket, but that was over a week ago | 23:17 |
| sdake | yup | 23:17 |
| SamYaple | like i said probably outdated image/repo | 23:17 |
| sdake | he has an updated repo - i took control of his machine | 23:17 |
| sdake | and its not out of date | 23:17 |
| sdake | check script isn't working | 23:17 |
| *** pbourke- has joined #kolla | 23:18 | |
| SamYaple | im fairly certain you are wrong and I could tell you the issue in about 5 seconds | 23:18 |
| sdake | 1 | 23:18 |
| sdake | 2 | 23:18 |
| sdake | 3 | 23:18 |
| sdake | 4 | 23:18 |
| sdake | 5 | 23:18 |
| SamYaple | that was 6 seconds | 23:18 |
| SamYaple | hey dave-mccowan are you duplicating the kolla-ansible script? | 23:19 |
| dave-mccowan | yea, to address code review comments from patch 1 to keep this out of kolla-ansible | 23:20 |
| SamYaple | dave-mccowan: yea thats not going to work | 23:20 |
| SamYaple | let me read the comment again | 23:20 |
| SamYaple | but this duplication is a no go | 23:20 |
| dave-mccowan | i'm open to anything. i figured this way would be extensible, in case there were other things in the future that didn't belong in kolla-ansible. | 23:21 |
| SamYaple | no dave-mccowan i dont see why this shouldnt be in kolla-ansible | 23:21 |
| dave-mccowan | works for me. | 23:22 |
| *** haplo37 has quit IRC | 23:25 | |
| *** pbourke- has quit IRC | 23:26 | |
| SamYaple | sdake: just fyi if keepalived worked on the host you were most definetely not running the latest harpxoy and keepalived code | 23:28 |
| sdake | so something odd happened | 23:28 |
| SamYaple | you were running mixed versions, thats what happened | 23:28 |
| *** JoseMello has quit IRC | 23:28 | |
| sdake | after he ran keepalived on the host, and then stopped keepalived, a deployment worked | 23:28 |
| SamYaple | no it didnt work like you think | 23:29 |
| SamYaple | he stopped keepalived and the vip stuck around | 23:29 |
| sdake | suspicion is the arp table held he ip address still in cache | 23:29 |
| SamYaple | nope not it at all | 23:29 |
| SamYaple | this is old out dated code | 23:29 |
| sdake | what is old and oudated? | 23:29 |
| SamYaple | what you were running and building | 23:29 |
| sdake | it is definately master | 23:29 |
| SamYaple | you may have git cloned master, but if there was an old pip install then you likely werent building and/or deploying master | 23:30 |
| SamYaple | my guess is you built new containers, but deployed old code | 23:30 |
| sdake | i removed the pip installed version | 23:30 |
| SamYaple | just a guess though | 23:30 |
| SamYaple | but again, 100% positive you were running old code somewhere if you say keepalived worked on the host | 23:31 |
| sdake | keepalived worked on bare metal | 23:31 |
| SamYaple | yup then you certainly were running old code womewhere | 23:31 |
| sdake | i think you assume without a basis | 23:31 |
| sdake | the code is not old | 23:31 |
| SamYaple | alright. well see | 23:32 |
| sdake | the code is master, the containers are master, i built them myself | 23:32 |
| SamYaple | i bet it will be like all those other time we disagreed and i showed you why | 23:32 |
| sdake | the one time you mean regarding ntpd? | 23:32 |
| sdake | you win on ntpd | 23:32 |
| sdake | fix this one :) | 23:32 |
| SamYaple | and heat config | 23:32 |
| sdake | SamYaple do you ahve a windows machine around? | 23:32 |
| SamYaple | I do not but as it turns out i need one for this stupid switch | 23:33 |
| SamYaple | actually you know what i think i do... | 23:33 |
| *** yuanying has joined #kolla | 23:33 | |
| *** yuanying has quit IRC | 23:34 | |
| *** yuanying has joined #kolla | 23:34 | |
| dtturner | Hi folks. Has anyone ever seen the following error during keystone bootstrap? http://paste.openstack.org/show/488290/ | 23:35 |
| dtturner | I'm testing a multinode deploy using images build from trunk last week (kolla==2.0.0.0b3.dev219). | 23:35 |
| sdake | ok SamYaple | 23:36 |
| sdake | here is what I think happened | 23:36 |
| sdake | your right he had old images | 23:36 |
| sdake | on the deployed target cache | 23:36 |
| sdake | during deployment, new images are not deployed | 23:36 |
| sdake | so we need to add cleanup-images to the docs | 23:37 |
| sdake | so the registry doesn't deploy new images if images already exists | 23:37 |
| SamYaple | nope | 23:37 |
| SamYaple | never has | 23:37 |
| SamYaple | not even the ansible docker module | 23:38 |
| sdake | ya i think everyone thinks they do | 23:38 |
| SamYaple | its never worked like that | 23:38 |
| sdake | for multinode | 23:38 |
| SamYaple | in fact the pull playbooks were only needed _because_ its never worked like that | 23:38 |
| *** achanda has quit IRC | 23:38 | |
| SamYaple | but anyway. yea. old code | 23:39 |
| *** salv-orl_ has quit IRC | 23:39 | |
| limamauricio_ | SamYaple, i'm thinking revert my patch and create a new one to solve the problem of Manila, neutron agent and openvswitch | 23:42 |
| SamYaple | limamauricio_: what is the reason a revert is needed? | 23:42 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add NB about cleanup-images https://review.openstack.org/284991 | 23:43 |
| limamauricio_ | I think it was very mixed, neutron role, manila role | 23:43 |
| sdake | limamauricio_ don't forget our friends linuxbridge ;-) | 23:44 |
| limamauricio_ | yes yes | 23:44 |
| limamauricio_ | Manila will switch to using linux bridge by default | 23:44 |
| SamYaple | limamauricio_: kolla is openvswitch by default | 23:45 |
| sdake | enough fun for one day, I'm off | 23:45 |
| SamYaple | sdake: all solved then? | 23:45 |
| limamauricio_ | yes | 23:45 |
| sdake | yup look at last review | 23:45 |
| SamYaple | ok ill review in a bit | 23:45 |
| SamYaple | setting up 10gb switch | 23:45 |
| limamauricio_ | Manila will still support ovs, just will not be the default | 23:46 |
| sdake | did you get that netgear brand? | 23:46 |
| SamYaple | yea | 23:46 |
| SamYaple | same one you have | 23:46 |
| sdake | i didn't do anything to config it | 23:46 |
| sdake | just plugged it in | 23:46 |
| SamYaple | net to get to web interface to lock down vlans | 23:46 |
| SamYaple | need* | 23:46 |
| *** vhosakot has quit IRC | 23:47 | |
| sdake | oh I thought it was completely dumb didn't knwo it had vlans | 23:47 |
| limamauricio_ | I'll need a bit of your help SamYaple, i do not know if my last updates in the patch are the best way; | 23:47 |
| sdake | limamauricio_ we typically dont revert patches that have gone through a thorough review | 23:47 |
| SamYaple | its got a full on web interface with LACP LAG and al lkinds of stuff sdake | 23:47 |
| sdake | if there is a bug in the implementation, fix it please with a new patch ;) | 23:47 |
| sdake | cool | 23:47 |
| sdake | i'll have to check it out | 23:47 |
| sdake | when i gets plugged back in | 23:48 |
| limamauricio_ | ok sdake | 23:48 |
| limamauricio_ | but if you put a check to check that enable_manila = yes, I believe that this will ensure that neutron agents and openvswitch will only be deployed in the storage node is enable_manila = yes | 23:49 |
| limamauricio_ | what do you think SamYaple? ^^ | 23:49 |
| *** achanda has joined #kolla | 23:49 | |
| limamauricio_ | your opinion is important, I'm looking for the best way to do this | 23:50 |
| SamYaple | limamauricio_: ill look at the review later | 23:55 |
| SamYaple | should be simple enough | 23:55 |
| *** dtturner has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!