| *** eghobo_ has joined #tripleo | 00:06 | |
| *** daneyon_ has joined #tripleo | 00:07 | |
| *** eghobo has quit IRC | 00:09 | |
| *** daneyon has quit IRC | 00:10 | |
| *** daneyon_ has quit IRC | 00:16 | |
| *** david-lyle has quit IRC | 00:16 | |
| *** eghobo_ has quit IRC | 00:19 | |
| *** daneyon has joined #tripleo | 00:20 | |
| *** rainya has quit IRC | 00:24 | |
| *** eghobo has joined #tripleo | 00:30 | |
| *** daneyon has quit IRC | 00:33 | |
| *** akrivoka has joined #tripleo | 00:35 | |
| *** julim has joined #tripleo | 00:40 | |
| *** akrivoka_ has joined #tripleo | 00:48 | |
| *** akrivoka has quit IRC | 00:52 | |
| *** julim has quit IRC | 00:56 | |
| StevenK | greghaynes: The failure on 177002 is hilarious | 01:02 |
|---|---|---|
| *** MasterPiece has quit IRC | 01:03 | |
| openstackgerrit | Jeremy Stanley proposed openstack/diskimage-builder: Switch from MySQL-python to PyMySQL https://review.openstack.org/184372 | 01:07 |
| *** eghobo has quit IRC | 01:09 | |
| openstackgerrit | Jeremy Stanley proposed openstack/os-apply-config: Switch from MySQL-python to PyMySQL https://review.openstack.org/184390 | 01:09 |
| openstackgerrit | Jeremy Stanley proposed openstack/tripleo-heat-templates: Switch from MySQL-python to PyMySQL https://review.openstack.org/184398 | 01:09 |
| openstackgerrit | Jeremy Stanley proposed openstack/tripleo-image-elements: Switch from MySQL-python to PyMySQL https://review.openstack.org/184399 | 01:09 |
| openstackgerrit | Jeremy Stanley proposed openstack/tuskar: Switch from MySQL-python to PyMySQL https://review.openstack.org/184402 | 01:09 |
| lifeless | wooo | 01:10 |
| *** akrivoka_ has quit IRC | 01:10 | |
| *** julim has joined #tripleo | 01:12 | |
| *** Marga_ has quit IRC | 01:13 | |
| *** masco has joined #tripleo | 01:14 | |
| openstackgerrit | Merged openstack/diskimage-builder: Export DIB_RELEASE in centos https://review.openstack.org/179433 | 01:29 |
| *** masco has quit IRC | 01:30 | |
| *** eghobo has joined #tripleo | 01:53 | |
| *** rainya has joined #tripleo | 02:06 | |
| *** rainya has quit IRC | 02:06 | |
| *** untriaged-bot has joined #tripleo | 03:00 | |
| untriaged-bot | Untriaged bugs so far: | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1455175 | 03:00 |
| openstack | Launchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107) | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1449852 | 03:00 |
| openstack | Launchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87) | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1449854 | 03:00 |
| openstack | Launchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87) | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1454803 | 03:00 |
| openstack | Launchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1454802 | 03:00 |
| openstack | Launchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/tuskar/+bug/1452752 | 03:00 |
| openstack | Launchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed] | 03:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1456648 | 03:00 |
| openstack | Launchpad bug 1456648 in diskimage-builder "rhel-common element tries to attach when using activation key" [Undecided,In progress] - Assigned to John Trowbridge (trown) | 03:00 |
| *** untriaged-bot has quit IRC | 03:01 | |
| *** eghobo_ has joined #tripleo | 03:24 | |
| *** eghobo has quit IRC | 03:27 | |
| *** yog_ has joined #tripleo | 03:43 | |
| *** eghobo_ has quit IRC | 03:48 | |
| *** eghobo has joined #tripleo | 03:49 | |
| *** eghobo_ has joined #tripleo | 04:44 | |
| *** eghobo has quit IRC | 04:47 | |
| *** chlong_ has quit IRC | 05:03 | |
| *** eghobo has joined #tripleo | 05:07 | |
| *** eghobo_ has quit IRC | 05:08 | |
| *** ParsectiX has quit IRC | 05:09 | |
| *** chlong_ has joined #tripleo | 05:16 | |
| *** eghobo has quit IRC | 05:17 | |
| *** eghobo has joined #tripleo | 05:23 | |
| *** rlandy has joined #tripleo | 05:47 | |
| *** Marga_ has joined #tripleo | 05:48 | |
| *** Marga_ has quit IRC | 05:49 | |
| *** Marga_ has joined #tripleo | 05:49 | |
| *** eghobo has quit IRC | 05:50 | |
| *** yog_ has quit IRC | 05:52 | |
| *** yog_ has joined #tripleo | 05:59 | |
| *** adrianopetrich has joined #tripleo | 05:59 | |
| *** mcornea has joined #tripleo | 06:07 | |
| *** ukalifon1 has joined #tripleo | 06:08 | |
| *** chlong_ has quit IRC | 06:09 | |
| *** MasterPiece has joined #tripleo | 06:16 | |
| *** dasm_ is now known as dasm | 06:18 | |
| dasm | o/ | 06:19 |
| dasm | good morning CET | 06:19 |
| dasm | and good * | 06:19 |
| marios | o/ | 06:19 |
| marios | good * right back atcha | 06:20 |
| * marios espresso topup | 06:20 | |
| openstackgerrit | Merged openstack/tripleo-common: Scale out heat stack https://review.openstack.org/173283 | 06:22 |
| *** jprovazn has joined #tripleo | 06:28 | |
| *** chlong has joined #tripleo | 06:31 | |
| chlong | mkeir, restarting the dhcpd on the BOS prod foreman proxy did not fix it as hoped. I'm digging a bit deeper to see what I can find. Could be file permissions on dhcp conf files. | 06:32 |
| chlong | hcp dir owned root:root that's not right should be root:foreman-proxy. Hmm. | 06:32 |
| *** aufi has joined #tripleo | 06:34 | |
| *** dasm has quit IRC | 06:34 | |
| *** dasm has joined #tripleo | 06:35 | |
| *** yog_ has quit IRC | 06:40 | |
| *** MasterPiece has quit IRC | 06:44 | |
| *** mestery has joined #tripleo | 06:48 | |
| *** gfidente has joined #tripleo | 06:52 | |
| *** gfidente has joined #tripleo | 06:52 | |
| marios | gfidente: are my eyes deceiving me? https://review.openstack.org/#/c/184263/ | 06:56 |
| gfidente | hey morning :) | 06:57 |
| marios | o/ | 06:57 |
| *** shardy_ has joined #tripleo | 06:57 | |
| gfidente | it's green yes! | 06:58 |
| *** shardy has quit IRC | 06:59 | |
| *** yog_ has joined #tripleo | 07:00 | |
| *** radek_ has joined #tripleo | 07:00 | |
| *** shardy_ has quit IRC | 07:02 | |
| *** radek_ has quit IRC | 07:02 | |
| *** ifarkas has joined #tripleo | 07:03 | |
| *** shardy has joined #tripleo | 07:03 | |
| *** radek_ has joined #tripleo | 07:03 | |
| gfidente | mandre, we should merge all the tree though, spare some +As | 07:03 |
| gfidente | at which point, we should have the infra services and the database populated at closure of step 2 | 07:07 |
| gfidente | and without risking races during infra services setup | 07:07 |
| marios | gfidente: yeah but what i don't get is why only that latest review has green on pupet job | 07:07 |
| marios | gfidente: will rechecks clear it (doesn't make sense since it has the others as dependencies) | 07:08 |
| marios | gfidente: my latest run just failed... going to poke at the logs some more | 07:08 |
| marios | gfidente: finish your breakfast man and se you in a bit | 07:08 |
| gfidente | because without ordered=true | 07:08 |
| gfidente | the rabbitmq nodes were isolated | 07:08 |
| gfidente | so the ones which 'fix' things are the sync_db usage for keystone and cinder | 07:09 |
| gfidente | rhe rabbitmq clients switched to list of hosts | 07:09 |
| gfidente | and the ordered=true clone param | 07:09 |
| marios | gfidente: ok thanks (damnit still seeing May 20 03:05:52 localhost os-collect-config: ^[[1;31mError: Unable to connect to mongodb server! (192.0.2.13:27017)^[[0m | 07:10 |
| gfidente | yeah there are a bunch of issues still happening at times | 07:10 |
| gfidente | actually rabbitmq itself didn't happen to be isolated all the times | 07:11 |
| gfidente | again probably due to racing when resource was added | 07:11 |
| gfidente | there is the timeout with httpd start as well, sometimes happening in CI | 07:11 |
| mandre | oh man, I'll soon have to change my nick ;-) | 07:11 |
| marios | gfidente: so then those reviews should geta clean puppet ha run ? (i mean like the sync_db stuff) | 07:11 |
| marios | mandre: sorry :( | 07:11 |
| gfidente | they won't because they miss to fix the others | 07:11 |
| mandre | no worries :) | 07:12 |
| mandre | have a good night | 07:12 |
| gfidente | mandre ^_^ | 07:12 |
| marios | mandre: its funny, my name is actually marios andreou so it is quite a coincidence | 07:12 |
| gfidente | we trying to get you into the conversation! | 07:12 |
| marios | night | 07:12 |
| gfidente | marios, so for instance sync_db was still not passing because computes couldn't reconnect to rabbit | 07:13 |
| gfidente | and changing clients with list of hosts was still not passing because nodes were isolated | 07:13 |
| *** panda has quit IRC | 07:13 | |
| gfidente | and yet some jobs hit the horizon startup timeout | 07:14 |
| mandre | gfidente, yeah, I'm not sure you'll get many insightful comments… but hey, worth a try | 07:14 |
| gfidente | so no, you will only get green on the topmost | 07:14 |
| gfidente | I'll get comments on what? | 07:14 |
| mandre | getting me involved in the conversation | 07:15 |
| *** panda has joined #tripleo | 07:15 | |
| gfidente | mandre, where are you based btw? | 07:15 |
| mandre | I'm usually in Japan, but in Vancouver for the week | 07:16 |
| gfidente | auch | 07:16 |
| *** jistr has joined #tripleo | 07:18 | |
| gfidente | jistr, morning :) | 07:19 |
| jistr | gfidente: morning :) | 07:19 |
| *** mestery has quit IRC | 07:19 | |
| gfidente | so I need to get coffee still | 07:19 |
| gfidente | but today I realized it is time to figure what to do with steps 3/4 now | 07:20 |
| gfidente | we can't add resources to pacemaker there because pacemaker_master will try to clone them on other nodes ... which haven't runned step 5 yet, so would launch services unconfigured! | 07:20 |
| *** panda has quit IRC | 07:21 | |
| gfidente | *haven't runned step 4 yet | 07:21 |
| gfidente | one option seems to be | 07:22 |
| gfidente | to move openstack configuration into step 1 (or step 2) forcing sync_db to false | 07:22 |
| *** panda has joined #tripleo | 07:23 | |
| *** regebro has joined #tripleo | 07:23 | |
| gfidente | another to do the openstack configuration in step 3 for all nodes, setting sync_db to true only for master and finally add services to pacemaker in step 4 (this seems the cleanest to me) | 07:24 |
| lsmola_ | gfidente: hello | 07:25 |
| gfidente | jistr, ^^ this one seems to be easy to accomplish by | 07:25 |
| gfidente | 1) removing and $pacemaker_master | 07:25 |
| gfidente | 2) making step 4 into its own condition where we add the pacemaker resources | 07:26 |
| gfidente | jistr, ^^ I'd like to attempt this if sounds good to you | 07:26 |
| lsmola_ | gfidente: quick question, there will be one ceph role for cinder,glance and swift backends? or multiple roles? | 07:26 |
| gfidente | lsmola_, morning, only one | 07:26 |
| lsmola_ | gfidente: ok | 07:26 |
| lsmola_ | gfidente: do we plan to use the old cinder storage and swift storage roles in the future? | 07:27 |
| jistr | gfidente: yeah the second looks cleaner to me as well. The first option might be a little bit hard to do while keeping the code looking nice (or at least as it is now :D ). lemme look at step 3/4 for a bit | 07:27 |
| gfidente | lsmola_, I think so, yes | 07:27 |
| lsmola_ | gfidente: will they work together with the ceph role? or they are mutually exclusive? | 07:28 |
| gfidente | together | 07:28 |
| lsmola_ | gfidente: ok | 07:28 |
| gfidente | cinder nodes provide additional space on LVM | 07:28 |
| lsmola_ | gfidente: thank you | 07:28 |
| gfidente | ceph nodes provide additional replicated space | 07:28 |
| gfidente | jistr, it would also be just a replica of the 1/2 | 07:29 |
| jistr | marios: morning :) would you have time to look at https://review.openstack.org/#/c/184059/2 and the follow up patches please? | 07:30 |
| gfidente | jistr, by moving everything in step 1 instead, we could make deployment faster | 07:30 |
| gfidente | removing entirely step 4 | 07:30 |
| gfidente | so it has some tech advantage | 07:30 |
| marios | jistr: sure man. i started doing that but wanted to try first. i started a run with that review with the green ha | 07:32 |
| marios | jistr: hit mongodb issue (unable to connect)... am still poking. will land them in a bit | 07:33 |
| gfidente | jistr, the thing is | 07:34 |
| gfidente | as of today | 07:34 |
| gfidente | if we decouple step3 from step4 | 07:34 |
| gfidente | ha job will become red again | 07:34 |
| gfidente | until all services are started by pacemaker in step4 | 07:34 |
| gfidente | because they could try to start from step3 when pacemaker_master hasn't launched db_sync yet | 07:34 |
| *** radek_ has quit IRC | 07:36 | |
| *** chlong has quit IRC | 07:37 | |
| jistr | gfidente: re getting rid of step 4 eventually, i think we'd need some "require => " bits for that, because we need to ensure that db_sync goes prior to pacemaker::resource::service, like this: http://fpaste.org/223672/07682143/ | 07:42 |
| marios | roles=`tuskar role-list | grep OpenStack | awk '{print $2}'` | 07:43 |
| marios | tuskar plan-delete overcloud | 07:43 |
| marios | tuskar-delete-roles --config-file /etc/tuskar/tuskar.conf --uuids $roles | 07:43 |
| marios | sry :( | 07:43 |
| jistr | jistr: can you think of a better solution to get rid of step 4? i'm thinking maybe keeping step 4 would make things easier to manage | 07:44 |
| jistr | gfidente: ^ | 07:47 |
| *** Goneri has joined #tripleo | 07:48 | |
| jistr | gfidente: so eventually i think we should get to something like this rather, which seems cleaner http://fpaste.org/223674/21079861/ | 07:48 |
| jistr | gfidente: re non-pacemakerized services starting from step 3 causing problems, lemme think of that | 07:49 |
| gfidente | jistr, yep that is what I would like to do as well | 07:53 |
| gfidente | seems cleaner | 07:54 |
| gfidente | for step 3 causing problems one dirty trick | 07:54 |
| gfidente | could be to use a *temporary* step | 07:54 |
| gfidente | the 5th, where we move the pacemaker resources addition | 07:55 |
| gfidente | but it would make dev even slower | 07:55 |
| gfidente | jistr, alternatively | 07:55 |
| jistr | gfidente: what about this http://fpaste.org/223675/14321085/ | 07:55 |
| jistr | gfidente: i could send a patch for that right now | 07:55 |
| jistr | gfidente: and we see if it breaks | 07:56 |
| marios | May 20 03:52:44 localhost galera(galera)[20303]: ERROR: local node <ov-raudx3cgdid-0-n3525qgaztiw-controller-vyox3uwuavjd.novalocal> must be a member of the wsrep_cluster_address <gcomm://ov-raudx3cgdid-0-n3525qgaztiw-controller-vyox3uwuavjd>to start this galera instance | 07:56 |
| marios | latest one i am seeing. is this addressed by something already i didn't grab | 07:57 |
| marios | mostly rhetoric don't mean to interrup | 07:57 |
| jistr | marios: not sure. So this failed the deployment? Or did it recover? | 07:57 |
| gfidente | jistr, that is, we remove L10 and L11 when the service is added to pacemakre? | 07:58 |
| jistr | marios: we've been seeing a galera error but pacemaker deals with that and everything works out fine eventually | 07:58 |
| gfidente | jistr, actually not remove but enforce falce | 07:58 |
| gfidente | *false | 07:58 |
| jistr | gfidente: we make those to lines => false and add the pacemaker resource to step 4, yes | 07:58 |
| jistr | s/to/two/ | 07:59 |
| gfidente | jistr, ack, like it | 07:59 |
| jistr | gfidente: cool, i'm on it | 07:59 |
| gfidente | seems to work also | 07:59 |
| gfidente | non_pcmk_start should be L5 | 07:59 |
| marios | jistr: still waitin actually | 07:59 |
| gfidente | actually no, it can be L2 | 08:00 |
| jistr | gfidente: only for keystone it will break the db sync if that patch hasn't been merged yet | 08:01 |
| jistr | gfidente: we need to bug some puppet-core about it | 08:01 |
| gfidente | yeah | 08:01 |
| gfidente | sbadia! :) | 08:02 |
| gfidente | https://review.openstack.org/#/c/180565/ << shameless please | 08:02 |
| jistr | gfidente: so i'll actually make it L5 for now, which will keep the keystone db sync working | 08:02 |
| gfidente | jistr, ack | 08:03 |
| *** regebro has quit IRC | 08:05 | |
| marios | eventual fail with May 20 04:02:48 localhost os-collect-config: Error: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) | 08:05 |
| *** regebro has joined #tripleo | 08:05 | |
| *** mmagr has joined #tripleo | 08:13 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Prepare for adding OpenStack services to Pacemaker https://review.openstack.org/184436 | 08:32 |
| jistr | gfidente: ^ just code to look at now, didn't run it yet, i bet i made some typo or forgot something somewhere :) | 08:32 |
| gfidente | jistr, I think it is nice | 08:37 |
| gfidente | you know what I wanted to do | 08:37 |
| gfidente | to check if/what services are running on non-master at the end of step 3 | 08:37 |
| jistr | gfidente: you mean in current state or with that patch ^ ? | 08:38 |
| gfidente | jistr, so I will test hacking a little the condition on L46 and see what we left 'running' | 08:38 |
| gfidente | jistr, with the patch! | 08:38 |
| jistr | gfidente: ack ;) | 08:38 |
| gfidente | I realized we do few uncontrolled imports | 08:39 |
| marios | gfidente: jistr ok guyes, you've been running this since uyesterday right? no need to duplicate. i will start landing things (yet to workout for me...) where should i start? | 08:39 |
| gfidente | memcache is one | 08:39 |
| marios | starting under https://review.openstack.org/#/c/184263 | 08:39 |
| jistr | marios: yeah that's correct (or from https://review.openstack.org/#/c/184059/2 upwards as you prefer) | 08:40 |
| marios | jistr:thx | 08:41 |
| jistr | gfidente: just looked into puppet-memcached https://github.com/saz/puppet-memcached/blob/master/manifests/init.pp | 08:43 |
| jistr | gfidente: looks like there's no way to write config without starting it | 08:44 |
| *** pino|work has joined #tripleo | 08:44 | |
| gfidente | jistr, heh :( | 08:44 |
| * pino|work waves to the non-summit people | 08:45 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Ensure sync_db is consumed by Keystone and Cinder classes https://review.openstack.org/184059 | 08:45 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move NTP and Memacache respectively into step 1 and step 2 https://review.openstack.org/184063 | 08:48 |
| openstackgerrit | Pino Toscano proposed openstack/diskimage-builder: Cleanup /tmp in the guest https://review.openstack.org/177325 | 08:49 |
| *** Marga_ has quit IRC | 08:49 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Enable VIPs via Pacemaker from step 2 instead of step 1 https://review.openstack.org/184072 | 08:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Consolidate use of $pacemaker_master in step 2 https://review.openstack.org/184078 | 08:55 |
| *** julim has quit IRC | 08:55 | |
| *** yog_ has quit IRC | 08:58 | |
| *** untriaged-bot has joined #tripleo | 09:00 | |
| untriaged-bot | Untriaged bugs so far: | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1455175 | 09:00 |
| openstack | Launchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107) | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1449852 | 09:00 |
| openstack | Launchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87) | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1449854 | 09:00 |
| openstack | Launchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87) | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1454803 | 09:00 |
| openstack | Launchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1454802 | 09:00 |
| openstack | Launchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/tuskar/+bug/1452752 | 09:00 |
| openstack | Launchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed] | 09:00 |
| untriaged-bot | https://bugs.launchpad.net/diskimage-builder/+bug/1456648 | 09:00 |
| openstack | Launchpad bug 1456648 in diskimage-builder "rhel-common element tries to attach when using activation key" [Undecided,In progress] - Assigned to John Trowbridge (trown) | 09:00 |
| *** untriaged-bot has quit IRC | 09:01 | |
| jistr | marios: thx! we should wait for CI on this one https://review.openstack.org/#/c/181081/7 | 09:03 |
| jistr | and after that gets +A, we should have a green HA CI | 09:03 |
| jistr | marios: there might be some intermittently appearing bugs that we haven't hit yet, it's probable that what you hit wasn't an issue in your env, but something which is indeed a problem but others just didn't hit it yet | 09:04 |
| marios | jistr: right. i am away tomorrow, and have couple tasks outstanding so thought id try again later with master | 09:13 |
| marios | (and finish up other stuff meantime) | 09:13 |
| openstackgerrit | Jan Provaznik proposed openstack/tripleo-common: Scale down heat stack https://review.openstack.org/184442 | 09:15 |
| *** jtomasek has joined #tripleo | 09:16 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Prepare for adding OpenStack services to Pacemaker https://review.openstack.org/184436 | 09:18 |
| *** yog_ has joined #tripleo | 09:19 | |
| openstackgerrit | Pino Toscano proposed openstack/diskimage-builder: Cleanup /tmp in the guest https://review.openstack.org/177325 | 09:25 |
| *** mmagr is now known as mmagr|afk | 09:30 | |
| *** ishant has joined #tripleo | 09:41 | |
| *** yog_ has quit IRC | 09:43 | |
| gfidente | we should at least try to agree on the easy things though : http://paste.openstack.org/show/229209/ | 10:08 |
| *** mmagr|afk is now known as mmagr | 10:39 | |
| jistr | gfidente: i view them as different things but it might be that we're not using them well at some spots (and sometimes the line is blurred). E.g. try googling for "todo vs fixme" | 10:43 |
| gfidente | I was joking! :) | 10:44 |
| jistr | gfidente: duh :D | 10:45 |
| gfidente | marios, jistr so it looks like the mysql issue is caused by galera-ready | 10:51 |
| gfidente | but I think we can remove that, and also mongodb-ready | 10:51 |
| gfidente | by moving the dependencies into subsequent step | 10:51 |
| gfidente | I'm building something on top of https://review.openstack.org/184436 | 10:51 |
| jistr | gfidente: cool. Btw i've had a successful deployment with https://review.openstack.org/#/c/184436/ | 10:52 |
| gfidente | jistr, I faced the mysql issue that marios mentioned | 10:52 |
| jistr | huh the fedora job breakage on ^^ was probably not a one time thing http://goodsquishy.com/downloads/tripleo-jobs.html | 11:10 |
| gfidente | yeah but looks like networking stuff | 11:11 |
| jistr | hmm looks like cache problem maybe | 11:12 |
| jistr | it's looking for dl.fedoraproject.org/pub/fedora/linux/updates/20/x86_64/glusterfs-3.5.3-1.fc20.x86_64.rpm | 11:12 |
| jistr | but the present one is glusterfs-3.5.3-2.fc20.x86_64.rpm | 11:12 |
| gfidente | jistr, so I did that attempt with >=5 | 11:12 |
| gfidente | going to add comments on the review | 11:12 |
| jistr | gfidente: cool, thx | 11:12 |
| gfidente | but I get mongodb, which is expected | 11:12 |
| gfidente | I get memcached which also is expected | 11:13 |
| *** dguerri is now known as dguerri`away | 11:13 | |
| gfidente | I get redis, expected | 11:13 |
| gfidente | the httpd probably is just enabled by default | 11:13 |
| *** panda has quit IRC | 11:20 | |
| *** panda has joined #tripleo | 11:21 | |
| gfidente | jistr, but I also get all the swift services started | 11:21 |
| jistr | gfidente: ah yeah i didn't add swift yet. It doesn't keep any data in galera so it shouldn't cause problems that we changed the step 3/4 step into just step 3 on all nodes | 11:23 |
| jistr | gfidente: i can look into it too though | 11:24 |
| jistr | gfidente: so i was looking at the CI problem but i'm unsure if that's more on our side or on fedora side | 11:24 |
| jistr | would like to be more sure before i go bug the fedora folks about it | 11:24 |
| * jistr tries to grep F20 repodata for something relevant | 11:27 | |
| *** pino|work has quit IRC | 11:27 | |
| gfidente | jistr, yup I don't know if we want to disable all swift services just to make sure they don't race with pacemaker | 11:28 |
| jistr | gfidente: yeah eventually we should probably, but until we start poking with pacemaker + swift, it shouldn't break the gate. The rest of the services might break after i removed the "only run this in step 3 if you're bootstrap node" though. | 11:30 |
| jistr | @ ~/tmp/f20-repo-issue | 11:31 |
| jistr | $ cat e0c92b2837730abd1bf8140ab00cd8f3717725b9ec13e48f074bb28260c3d5a5-primary.xml | grep glusterfs-3.5.3-1 | wc -l | 11:31 |
| jistr | 0 | 11:31 |
| jistr | @ ~/tmp/f20-repo-issue | 11:31 |
| jistr | $ cat e0c92b2837730abd1bf8140ab00cd8f3717725b9ec13e48f074bb28260c3d5a5-primary.xml | grep glusterfs-3.5.3-2 | wc -l | 11:31 |
| jistr | 17 | 11:31 |
| jistr | ^^ it's probably not a fedora problem, unless it's a specific mirror which is broken | 11:31 |
| jistr | i mean out of date | 11:31 |
| jistr | and we keep downloading repodata from that one mirror and than trying to download packages from some other mirror | 11:32 |
| * jistr will not bug fedora people about this then | 11:32 | |
| * jistr looking at swift | 11:33 | |
| *** pino|work has joined #tripleo | 11:38 | |
| jistr | [root@ov-nfvvcm2vqit-0-6rqxyisdtcxd-controller-d7hip5ixd42h ~]# systemctl | grep openstack-swift | wc -l | 11:41 |
| jistr | 13 | 11:41 |
| jistr | ;D | 11:42 |
| *** thrash|g0ne is now known as thrash | 11:43 | |
| pino|work | what are those? | 11:45 |
| jistr | pino|work: http://fpaste.org/223744/22330143/raw/ | 11:45 |
| pino|work | nice | 11:45 |
| * pino|work wonders what happens if you turn some random ones off | 11:46 | |
| gfidente | jistr, I was also wondering if we shouldn't strip redis entirely | 11:47 |
| jistr | gfidente: so re swift -- i think we'll need to patch this to add enable and manage_service https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/all.pp | 11:48 |
| gfidente | also an option, but you can pass those to the three different modules already | 11:48 |
| jistr | gfidente: alternatively we could unwrap the whole thing into our manifest but that grows it unnecessarily | 11:49 |
| gfidente | so in the newer doc from beekof ceilometer is a/a with redis | 11:49 |
| jistr | gfidente: so we shouldn't drop redis then i guess | 11:50 |
| gfidente | yeah and spredzy added the manage_service there 7days ago :) | 11:51 |
| gfidente | https://github.com/arioch/puppet-redis/commit/aa13ac935693204972b7255233f13402634f256e | 11:51 |
| gfidente | so we're good as that is on his head :P | 11:51 |
| jistr | cool :D | 11:52 |
| gfidente | fine I'll play with the -ready thing, swift looks like we can wait, I would really only add a note there | 11:52 |
| gfidente | we need to unstuck CI to merge the remaining stuff though | 11:53 |
| jistr | gfidente: ack i'll add a note (not sure if TODO or FIXME :P) and also send a patch to puppet-swift fro that storage::all class | 11:53 |
| gfidente | nice :) | 11:53 |
| jistr | gfidente: i bet we just have to invalidate some cache for the CI problem to go away but i don't see any mention here https://wiki.openstack.org/wiki/TripleOPuppetCI | 11:59 |
| jistr | if we do cache fedora repos in the CI somewhere or not | 11:59 |
| openstackgerrit | Jiri Tomasek proposed openstack/tuskar-ui: Service config horizontal form tweaks https://review.openstack.org/184466 | 12:00 |
| gfidente | jistr, so the use of TODO(nick) comes from guidelines http://docs.openstack.org/developer/hacking/ | 12:00 |
| *** dguerri`away is now known as dguerri | 12:01 | |
| jistr | gfidente: yeah but we don't stick to it in t-h-t and also i think the (nick) part can be inferred from git blame | 12:01 |
| gfidente | you can't argue with guidelines! | 12:02 |
| jistr | gfidente: and also i made it a fixme :D :P | 12:02 |
| gfidente | and also, I think I used FIXME myself in t-h-t | 12:02 |
| gfidente | do you know why spredzy wanted to leave mysql as an option for ceilometer? | 12:04 |
| jistr | no i don't recall | 12:05 |
| jistr | hmm so that service_manage change for swift might be quite obtrusive | 12:22 |
| jistr | we define this https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/all.pp | 12:22 |
| jistr | which defines this (one per store) https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/server.pp | 12:23 |
| jistr | which *includes* the end class https://github.com/stackforge/puppet-swift/blob/master/manifests/storage/server.pp#L77 | 12:23 |
| *** lazy_prince has joined #tripleo | 12:24 | |
| jistr | so i'd have to change that include on L77 to a resource-like definition | 12:24 |
| jistr | i bet this would be rejected upstream | 12:24 |
| jistr | because then it prevents a resource-like definition from elsewhere | 12:24 |
| jistr | so we'll need to add those classes to our manifest indeed | 12:25 |
| jistr | (not via unwrapping the storage::all class though, as i thought at first) | 12:25 |
| jistr | gfidente: ^ | 12:25 |
| gfidente | jistr, I'd be happy to leave this entirely to you :)\ | 12:26 |
| gfidente | I +2 the change, if you really wanted to do something about it now, I would probably pass the enable/manage_service to all classes like for ceilometer and as per gerrit comment | 12:26 |
| jistr | gfidente: yeah, just wanted to explain why i don't go the way we discussed previously | 12:26 |
| *** jayg|g0n3 is now known as jayg | 12:28 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Prepare for adding OpenStack services to Pacemaker https://review.openstack.org/184436 | 12:33 |
| gfidente | I am not having a nice time with the galera-ready thing | 12:34 |
| jistr | lemme get some tea and i'll be back | 12:36 |
| gfidente | actually I think the solution is closer than I suspected | 12:38 |
| *** trown|outttypeww is now known as trown | 12:39 | |
| *** dasm is now known as dasm|afk | 12:42 | |
| jistr | gfidente: to get back to that -- you wanted to remove galera-ready entirely? | 12:44 |
| gfidente | that and mongodb-ready and shift what depends on it in later steps | 12:44 |
| gfidente | but I think it is fair to keep a hard split between 2/3 where 3 is expecting infra ready, including the database to exist | 12:45 |
| gfidente | so I am not removing them anymore, not for now | 12:45 |
| gfidente | let me push one thing | 12:45 |
| jistr | gfidente: i've just realized that the step separation might not be sufficient there | 12:45 |
| gfidente | yeah because internode locks we still don't have :P | 12:45 |
| gfidente | ctrl1 could get to step3 before ctrl0 | 12:46 |
| gfidente | I think validation is the only thing which will save use from this :) | 12:46 |
| jistr | gfidente: also i think we actually need the galera-ready bit to ensure that galera is usable in the beginning of step 3. I think someone mentioned that getting galera synced can take time. So if we remove it we could end up with galera not being usable yet when nodes start running step 3. | 12:50 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Prepare for adding OpenStack services to Pacemaker https://review.openstack.org/184436 | 12:54 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: We only need galera-ready on the pacemaker_master node https://review.openstack.org/184476 | 12:54 |
| gfidente | daaaah I forgot to rebase again :() | 12:54 |
| jistr | hahaha | 12:55 |
| jistr | i'll re-push the one, no worries | 12:55 |
| jistr | gfidente: i have it locally | 12:55 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Prepare for adding OpenStack services to Pacemaker https://review.openstack.org/184436 | 12:55 |
| gfidente | yes please, I will rebase, I promise | 12:55 |
| gfidente | sorry :( | 12:56 |
| gfidente | I did the checkout before last update | 12:56 |
| gfidente | as last time | 12:56 |
| jistr | gfidente: yeah np, i update it too often :) | 12:56 |
| jistr | gfidente: btw the CI fixed itself it seems | 12:56 |
| jistr | the job which was running got past 7 minutes | 12:57 |
| * jistr rechecked https://review.openstack.org/#/c/181081/ | 12:57 | |
| *** chlong has joined #tripleo | 12:58 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: We only need galera-ready on the pacemaker_master node https://review.openstack.org/184476 | 12:59 |
| gfidente | jistr, can you check ^^ ? | 12:59 |
| *** dguerri is now known as dguerri`away | 13:03 | |
| jistr | gfidente: i think there's a small syntax error, otherwise it looks great | 13:07 |
| gfidente | jistr, you think it does? | 13:07 |
| jistr | gfidente: maybe the mongodb bit could be a separate patch | 13:07 |
| gfidente | I am unsure | 13:07 |
| gfidente | I can strip that out yes | 13:07 |
| gfidente | I am unsure because I wonder what would happen to the other nodes in step3? | 13:07 |
| gfidente | ah they don't run db_sync anywasy | 13:08 |
| jistr | gfidente: i think it should fly... only one ready wait + only one user creation | 13:08 |
| gfidente | database creation as well, but I just copy/pasted really | 13:08 |
| *** dguerri`away is now known as dguerri | 13:08 | |
| gfidente | fi | 13:10 |
| gfidente | ^_^ | 13:10 |
| gfidente | where did I take that from? | 13:11 |
| jistr | gfidente: so one thing i'm worried about for the future -- the database connection strings go through VIP/HAproxy | 13:11 |
| gfidente | yeah I was going to ping you on that | 13:12 |
| jistr | and when we add galera-monitor check condition in HAproxy as HA arch / Astapor has it | 13:12 |
| gfidente | bingo | 13:12 |
| jistr | HAproxy might not start balancing to Galera soon enough | 13:12 |
| jistr | and we'd run the DB syncs on 503 service unavailable | 13:12 |
| gfidente | yes so I tried to work on that yesterday | 13:13 |
| jayg | yeah, you don't want to do any of that until galera is ready and haproxy is running, that is what we do | 13:13 |
| gfidente | but it's very hard to do in a single step I think | 13:13 |
| openstackgerrit | Marios Andreou proposed openstack/tuskar: Alter the type for the contents column of stored_file https://review.openstack.org/184481 | 13:14 |
| jayg | yeah, since you have steps you might break it up, sure | 13:14 |
| *** chlong has quit IRC | 13:14 | |
| gfidente | jistr, unless, given we create file and xinetd on all nodes regardless, as it should be | 13:14 |
| jayg | get all the 'infrastructure' type services up and ready, then add openstack services | 13:15 |
| gfidente | we use '::hostname' when creating the database | 13:15 |
| gfidente | jayg, yes that is the separation we're aiming at for step2/step3 | 13:15 |
| gfidente | but we do database creation in step2 | 13:15 |
| gfidente | we might want to move that to step3 as an option | 13:16 |
| gfidente | but we'd have to make all db_sync depend on it then | 13:16 |
| jayg | I's have to look again, but do you already only run db_sync on pacemaker master? | 13:16 |
| jistr | gfidente: so what is the error you were trying to solve with moving the bits only to bootstrap host? | 13:16 |
| jistr | gfidente: is it race on creating user/database? | 13:16 |
| gfidente | nope it is the galera-ready failing on the non-master nodes | 13:16 |
| jistr | jayg: yeah | 13:16 |
| jistr | gfidente: did you find out why does it fail consistently? It should retry and succeed eventually... | 13:17 |
| gfidente | /Stage[main]/Main/Exec[galera-ready]/returns: ERROR 2002 | 13:20 |
| gfidente | (HY000): Can't connect to local MySQL server through s | 13:20 |
| gfidente | ocket '/var/lib/mysql/mysql.sock' (111 "Connection refused") | 13:20 |
| *** lblanchard has joined #tripleo | 13:21 | |
| gfidente | it does mention returned 1 instead 0 | 13:21 |
| jistr | hmm strange | 13:22 |
| *** ishant has quit IRC | 13:22 | |
| gfidente | ah there is the real failure | 13:22 |
| gfidente | Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' | 13:22 |
| gfidente | this is not in the -ready thing | 13:23 |
| gfidente | it could be the user creation attempting it | 13:23 |
| gfidente | output does not seem to tell what was calling it | 13:24 |
| jistr | yeah so maybe our problem is not as much mysql-ready as the clustercheck user creation being attempted from non-master? | 13:24 |
| * jistr looks into the code again | 13:24 | |
| gfidente | clustercheck user is only in _master | 13:24 |
| gfidente | ah indeed the error is also on _master | 13:25 |
| gfidente | -ready error is on non-master as well | 13:25 |
| jayg | check your xinetd galera-monitor file and make sure you have user/group | 13:25 |
| gfidente | (expected probably) | 13:25 |
| jayg | I had a slightly old version I was using with astapor yesterday and those were missing | 13:25 |
| jayg | (old version of puppet-xinetd) | 13:25 |
| jistr | so yeah we're attempting to use the database while it's not yet ready for use... but the galera-ready exec should signify *global* readiness of the galera cluster, not just the one node, right? and the user creation does depend on galera-ready | 13:26 |
| jistr | user/group is root/root | 13:27 |
| gfidente | jistr, can you spot anything else trying to use the db? | 13:28 |
| gfidente | the db creation has require | 13:28 |
| *** chlong has joined #tripleo | 13:31 | |
| jistr | nope i can't see anything | 13:31 |
| jistr | gfidente: could you post the whole os-collect-config log of bootstrap node somewhere? | 13:31 |
| jistr | for some reason i haven't hit that issue yet | 13:32 |
| gfidente | jistr, redeploying, marios hit that though | 13:32 |
| gfidente | can -ready be passing when nodes are not joined into the cluster yet? | 13:33 |
| gfidente | jistr, we could maybe extend the -ready check a little to see if cluster is up not only if db is writable | 13:35 |
| jistr | gfidente: that sounds great, if you know how to do that, please do it :) | 13:39 |
| gfidente | so let's split this eventually and the mongodb in different changes | 13:40 |
| jistr | gfidente: so looking at the clustercheck script, apart from read-only they also look at this https://github.com/olafz/percona-clustercheck/blob/master/clustercheck#L58 | 13:41 |
| * jistr knows 0 of galera though | 13:42 | |
| gfidente | yeah ideally we could just use clustercheck like astapor does for -ready | 13:43 |
| gfidente | this would force us fix the user/pass thing today | 13:43 |
| gfidente | but looks like worth it? | 13:43 |
| gfidente | jistr, okay before you kill me | 13:44 |
| jistr | yeah looks worth it i think | 13:44 |
| gfidente | why the hell do we want to make life complicated with the clustercheck user? | 13:44 |
| gfidente | we give root to clustercheck user, like the resource agent does | 13:45 |
| *** radek__ has joined #tripleo | 13:45 | |
| gfidente | deploy it and use it for -ready command | 13:45 |
| gfidente | ? | 13:45 |
| jistr | gfidente: ack from me... i don't know why we have that user | 13:46 |
| gfidente | I can see it is better to have one | 13:46 |
| gfidente | but makes things extremely complicated in our scenario | 13:46 |
| gfidente | well extremely for me at least | 13:46 |
| gfidente | let me try | 13:46 |
| jistr | https://github.com/redhat-openstack/astapor/blob/23d85c20b1a4dd2f9b12ef1dbcf9b27d1ff23df9/puppet/modules/quickstack/manifests/pacemaker/galera.pp#L177-L183 | 13:47 |
| gfidente | EXACTLY | 13:47 |
| gfidente | :) | 13:47 |
| gfidente | and it will make haproxy configuration more robust as well | 13:47 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Ensures mongodb configuration only happens if mongodb is needed https://review.openstack.org/184494 | 13:49 |
| *** shardy_ has joined #tripleo | 13:51 | |
| *** shardy has quit IRC | 13:52 | |
| gfidente | jistr, I would still move those into $pacemaker_master though | 13:52 |
| gfidente | no need to keep them out of scope | 13:52 |
| gfidente | unless we want other nodes to stop until galera is ready | 13:53 |
| gfidente | so I won't move it for now | 13:53 |
| jistr | gfidente: yeah let's not move them yet. At least we'll know that clustercheck works on all nodes. (We do want to stop nodes from progressing until galera is ready, but Heat will stop them before step 3 until bootstrap node is done.) | 13:54 |
| *** shardy_ has quit IRC | 13:56 | |
| *** shardy has joined #tripleo | 13:57 | |
| *** mcornea has quit IRC | 13:57 | |
| *** mcornea has joined #tripleo | 13:59 | |
| *** jprovazn has quit IRC | 14:00 | |
| *** mcornea_ has joined #tripleo | 14:05 | |
| pino|work | if i'm not mistaken, elements/redhat-common/finalise.d/01-clean-old-kernels.sh is never executed, because allowed_regex in dib-run-parts does not allow "." in the name... | 14:06 |
| *** shardy_ has joined #tripleo | 14:06 | |
| *** shardy has quit IRC | 14:08 | |
| *** mcornea has quit IRC | 14:09 | |
| *** dguerri is now known as dguerri`away | 14:10 | |
| *** dguerri`away is now known as dguerri | 14:10 | |
| *** shardy_ has quit IRC | 14:11 | |
| *** shardy has joined #tripleo | 14:12 | |
| *** yog_ has joined #tripleo | 14:18 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Use clustercheck script to control galera-ready https://review.openstack.org/184505 | 14:19 |
| gfidente | jistr, ^^ tested :) | 14:19 |
| *** julim has joined #tripleo | 14:20 | |
| gfidente | someone should fix sad status of snmpd too :) | 14:24 |
| gfidente | we still install the element and yet attempt configuration via puppet filling journal with snmpd errors | 14:24 |
| gfidente | marios, you around? | 14:25 |
| jistr | gfidente: looks great, but should we reduce the timeout for mongodb/galera? Maybe 20 minutes would do? In case something gets stuck, you don't want to keep waiting too long... | 14:25 |
| gfidente | ahahaha | 14:25 |
| gfidente | I actually agree | 14:25 |
| gfidente | but copied those from astapor | 14:25 |
| gfidente | :P | 14:25 |
| *** jprovazn has joined #tripleo | 14:26 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tuskar-ui: Service config horizontal form tweaks https://review.openstack.org/184466 | 14:26 |
| gfidente | jayg, is there a reason why you had 1h sync timeout? | 14:26 |
| *** jtomasek has quit IRC | 14:26 | |
| jayg | gfidente: reading back, galera/mongo sync in what area? | 14:27 |
| gfidente | jayg, clustercheck https://github.com/redhat-openstack/astapor/blob/23d85c20b1a4dd2f9b12ef1dbcf9b27d1ff23df9/puppet/modules/quickstack/manifests/pacemaker/galera.pp#L177-L183 | 14:27 |
| jayg | I dont think the hour there there is specific, just generous to cover slow machines | 14:28 |
| openstackgerrit | Pino Toscano proposed openstack/diskimage-builder: redhat-common: rename 01-clean-old-kernels.sh to drop .sh extension https://review.openstack.org/184509 | 14:28 |
| jayg | we use an hour consistently throughout quickstack | 14:29 |
| jayg | (well, for major things anyway) | 14:29 |
| gfidente | jistr, well I was thinking, if puppet fails heat reports immediately | 14:29 |
| gfidente | so the 1h timeout would only be hit if puppet is waiting on the exec, pacemaker thinks things are good, but clustercheck doesn't | 14:30 |
| gfidente | which looks like not terrible | 14:31 |
| gfidente | how about we tweak it down only if we hit that instead of playing it low in advance?\ | 14:31 |
| jistr | gfidente: well, i recall that timeout on wait-until-galera was quite a frequent failure scenario some time ago in Astapor | 14:32 |
| jistr | gfidente: let's bargain on 30 minutes then? :D | 14:32 |
| gfidente | okay but if they git blame me | 14:33 |
| gfidente | i'll give them your name | 14:33 |
| jistr | gfidente: ack :D | 14:33 |
| gfidente | btw is there a reson to use both timeout and tries? | 14:34 |
| jayg | timeout is how long to wait for the command to fail to return between tries | 14:35 |
| gfidente | ah so we can lower that to 30 | 14:36 |
| gfidente | makes sense? | 14:36 |
| jistr | ack | 14:36 |
| gfidente | which means, if sleep is 10 | 14:37 |
| gfidente | each cycle can take up to 40secs | 14:37 |
| jayg | correct | 14:38 |
| gfidente | so I would use 90 tries | 14:38 |
| gfidente | which means, maximum 1h of timed out attempts and maximum 15mins of successful but unsynced? | 14:39 |
| gfidente | too low | 14:39 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Use clustercheck script to control galera-ready https://review.openstack.org/184505 | 14:40 |
| gfidente | ^^ 30mins successful/unsynced attempts, 2h unsuccessful attempts | 14:40 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Use clustercheck script to control galera-ready https://review.openstack.org/184505 | 14:42 |
| *** yog_ has quit IRC | 14:43 | |
| jayg | gfidente: hmm, I notice in that patch you guys create the xinetd service manually, why not use galera::monitor for that? | 14:43 |
| gfidente | jayg, where is that? | 14:43 |
| gfidente | maybe we can add that on top | 14:44 |
| gfidente | and replace xinetd | 14:44 |
| gfidente | I still prefer smaller changes | 14:44 |
| jistr | gfidente: 180 $tries + 10 $try_sleep is perfect. I don't think we need to worry about $timeout too much, clustercheck is fast. | 14:45 |
| jayg | gfidente: yeah, not critial at all, I just noticed since I am having trouble with xinetd, might even make there be less in the puppet file here | 14:45 |
| jayg | it is part of puppet-galera module | 14:45 |
| gfidente | jayg, you've to join this and start pushing changes | 14:46 |
| gfidente | not joking :) | 14:46 |
| jistr | atm we don't use puppet-galera | 14:46 |
| gfidente | yeah but this I knew, we don't use puppet-galera | 14:46 |
| jayg | really? why not? | 14:46 |
| jayg | gfidente: and yeah, I agree, now that I have actually had a successful(ish) deploy, I may be able to do that, | 14:47 |
| jistr | would we use something more besides galera::monitor from it? | 14:48 |
| jistr | i recall someone being displeased with puppet-galera | 14:48 |
| jayg | however, my boss has told me I am 'officially' 100% on quickstack for kilo, so I am only at most 50% here, trying to help move things along by paying attention and testing | 14:48 |
| jayg | I don't know how exactly you are using galera without it, I would be interested to know who was displeased and why | 14:49 |
| jayg | maybe social? think he was having some ssl/ipv6 issue | 14:49 |
| jistr | yeah i think it was him actually :) | 14:49 |
| gfidente | jistr, can we +A https://review.openstack.org/#/c/181081/ ? | 14:50 |
| gfidente | so we unlock green CI because the clone_params one already has +A ~! | 14:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Provide RabbitMQ clients with a list of servers instead of VIP https://review.openstack.org/181081 | 14:51 |
| jistr | unlocked | 14:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Clone params for pacemaker rabbitmq resource https://review.openstack.org/184263 | 14:51 |
| jistr | \o/ | 14:51 |
| *** radek__ has quit IRC | 14:52 | |
| gfidente | HURRA | 14:53 |
| *** morazi has joined #tripleo | 14:54 | |
| jistr | so just to be clear -- i'm not against puppet-galera, i haven't looked at it yet. Just thinking we should weigh the benefits of it vs. the cost of depending on another puppet module, especially if there's not much community around it. | 14:55 |
| jayg | it is not high priority, since it seems you guys have managed to get galera running somehow without it, we can look later on | 14:56 |
| jayg | I was just surprised, that's all, thought everyone was using it | 14:56 |
| jayg | packstack uses galera::server, for instance | 14:57 |
| *** morazi_ has joined #tripleo | 14:57 | |
| jayg | even though they dont do HA | 14:57 |
| *** cwolferh has quit IRC | 14:58 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates: Move sysctl settings into hieradata https://review.openstack.org/184210 | 14:58 |
| gfidente | jistr, I rebased ^^ on top of the step 3/4 changes so we don't get merge conflicts | 14:59 |
| gfidente | I would still move the create/ready into $pacemaker_master scope though | 14:59 |
| gfidente | but interesting will be now to see some green jobs :) | 15:00 |
| *** untriaged-bot has joined #tripleo | 15:00 | |
| untriaged-bot | Untriaged bugs so far: | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1455175 | 15:00 |
| openstack | Launchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107) | 15:00 |
| untriaged-bot | https://bugs.launchpad.net/tripleo/+bug/1454802 | 15:00 |
| openstack | Launchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [High,New] | 15:00 |
| *** untriaged-bot has quit IRC | 15:00 | |
| *** morazi has quit IRC | 15:01 | |
| *** morazi__ has joined #tripleo | 15:10 | |
| *** ParsectiX has joined #tripleo | 15:10 | |
| *** ParsectiX has quit IRC | 15:11 | |
| *** morazi_ has quit IRC | 15:14 | |
| *** rlandy has quit IRC | 15:14 | |
| *** morazi_ has joined #tripleo | 15:17 | |
| *** morazi__ has quit IRC | 15:20 | |
| *** panda has quit IRC | 15:20 | |
| *** panda has joined #tripleo | 15:21 | |
| *** aufi has quit IRC | 15:24 | |
| *** ukalifon1 has quit IRC | 15:25 | |
| gfidente | jistr, stuff bitten by the mysql.socket issue http://logs.openstack.org/94/184494/1/check-tripleo/check-tripleo-ironic-overcloud-f20puppet-ha/6d76e4a/logs/ov-qm7mwaqcklq-0-5d5arig3i6yl-Controller_logs/os-collect-config.txt.gz | 15:27 |
| gfidente | so we might need a couple more merges still | 15:28 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-incubator: Wire --vlan-trunk-ids into devtest_testenv.sh https://review.openstack.org/183588 | 15:28 |
| gfidente | jistr, https://review.openstack.org/#/c/177735/ merge? | 15:38 |
| gfidente | I don't see why not :) | 15:39 |
| gfidente | GO! | 15:40 |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Set up corosync using hostnames rather than IPs https://review.openstack.org/177735 | 15:40 |
| *** alop has joined #tripleo | 15:42 | |
| *** mmagr is now known as para | 15:42 | |
| *** para is now known as Guest35369 | 15:43 | |
| *** morazi__ has joined #tripleo | 15:45 | |
| *** Guest35369 is now known as paramite | 15:49 | |
| *** morazi_ has quit IRC | 15:49 | |
| *** paramite has quit IRC | 15:51 | |
| *** julim has quit IRC | 15:51 | |
| *** paramite has joined #tripleo | 15:51 | |
| *** MasterPiece has joined #tripleo | 15:52 | |
| *** MasterPiece has quit IRC | 15:53 | |
| *** david-lyle has joined #tripleo | 15:54 | |
| *** mestery has joined #tripleo | 15:55 | |
| *** ifarkas has quit IRC | 15:55 | |
| gfidente | marios, are you around tomorrow? | 15:55 |
| *** mestery has quit IRC | 15:57 | |
| *** mestery has joined #tripleo | 15:57 | |
| *** mestery has quit IRC | 15:59 | |
| *** mestery has joined #tripleo | 15:59 | |
| *** eghobo has joined #tripleo | 16:02 | |
| *** paramite has quit IRC | 16:04 | |
| *** cwolferh has joined #tripleo | 16:04 | |
| *** akrivoka has joined #tripleo | 16:05 | |
| *** eghobo_ has joined #tripleo | 16:06 | |
| *** eghobo has quit IRC | 16:10 | |
| *** jcoufal has joined #tripleo | 16:12 | |
| *** morazi_ has joined #tripleo | 16:12 | |
| *** morazi__ has quit IRC | 16:17 | |
| jistr | o/ | 16:20 |
| *** jistr has quit IRC | 16:20 | |
| *** gfidente is now known as gfidente|afk | 16:24 | |
| *** mcornea_ has quit IRC | 16:28 | |
| *** jcoufal has quit IRC | 16:29 | |
| *** alop has quit IRC | 16:34 | |
| *** athomas has joined #tripleo | 16:36 | |
| *** jcoufal has joined #tripleo | 16:41 | |
| *** mestery has quit IRC | 16:47 | |
| *** Marga_ has joined #tripleo | 16:48 | |
| *** Goneri has quit IRC | 16:50 | |
| *** pino|work has quit IRC | 16:59 | |
| *** jdob has joined #tripleo | 17:00 | |
| *** Marga_ has quit IRC | 17:03 | |
| *** athomas has quit IRC | 17:03 | |
| openstackgerrit | Mayank Kumar proposed openstack/tripleo-image-elements: Option to configure virtual_route through keepalived https://review.openstack.org/184547 | 17:09 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Overcloud: bump HOT version to 2015-04-30 https://review.openstack.org/184550 | 17:11 |
| *** morazi_ has quit IRC | 17:13 | |
| *** zaneb has joined #tripleo | 17:13 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates: Move sysctl settings into hieradata https://review.openstack.org/184210 | 17:23 |
| *** jdob has quit IRC | 17:27 | |
| *** david-lyle has quit IRC | 17:29 | |
| *** weshay has joined #tripleo | 17:33 | |
| *** eghobo_ has quit IRC | 17:36 | |
| *** Marga_ has joined #tripleo | 17:38 | |
| *** MasterPiece has joined #tripleo | 17:38 | |
| *** jprovazn has quit IRC | 17:41 | |
| *** eghobo has joined #tripleo | 17:42 | |
| *** lazy_prince has quit IRC | 17:42 | |
| *** morazi has joined #tripleo | 17:46 | |
| *** trown is now known as trown|lunch | 17:49 | |
| *** julim has joined #tripleo | 17:54 | |
| *** alop has joined #tripleo | 17:56 | |
| *** akrivoka has quit IRC | 17:58 | |
| *** morazi has quit IRC | 18:00 | |
| *** cody-somerville has joined #tripleo | 18:03 | |
| *** cody-somerville has quit IRC | 18:03 | |
| *** cody-somerville has joined #tripleo | 18:03 | |
| *** weshay has quit IRC | 18:04 | |
| *** Marga_ has quit IRC | 18:05 | |
| *** zaneb has quit IRC | 18:05 | |
| *** Marga_ has joined #tripleo | 18:05 | |
| *** Marga_ has quit IRC | 18:10 | |
| *** julim has quit IRC | 18:19 | |
| *** julim has joined #tripleo | 18:32 | |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Overcloud: bump HOT version to 2015-04-30 https://review.openstack.org/184550 | 18:39 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Update neutron local_ip to use the tenant network https://review.openstack.org/178716 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add a network ports IP mapping resource https://review.openstack.org/178714 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated network ports to block storage roles https://review.openstack.org/180824 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated network ports to swift roles https://review.openstack.org/180823 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Switch net-config templates to use OS::stack_id https://review.openstack.org/184343 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated network ports to ceph roles https://review.openstack.org/180822 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated network ports to controller roles https://review.openstack.org/177846 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated network ports to compute roles https://review.openstack.org/180821 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add a ports (ip address) abstraction layer https://review.openstack.org/177845 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Add isolated net parameters to net-config stacks https://review.openstack.org/180820 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: Wire in optional network creation for overcloud https://review.openstack.org/177844 | 18:40 |
| openstackgerrit | Dan Prince proposed openstack/tripleo-heat-templates: os-net-config templates to configure vlans on bond https://review.openstack.org/184580 | 18:40 |
| *** dguerri is now known as dguerri`away | 18:42 | |
| *** weshay has joined #tripleo | 18:43 | |
| openstackgerrit | Merged openstack/tripleo-incubator: Add a puppet disk images YAML file https://review.openstack.org/181114 | 18:43 |
| *** jcoufal has quit IRC | 18:44 | |
| *** trown|lunch is now known as trown | 18:44 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: puppet: use a single image for CI https://review.openstack.org/177060 | 18:44 |
| *** morazi has joined #tripleo | 18:50 | |
| *** alop has quit IRC | 18:51 | |
| openstackgerrit | Merged openstack/diskimage-builder: Have simple-init regenerate ssh keys on boot https://review.openstack.org/181749 | 19:06 |
| openstackgerrit | Merged openstack/diskimage-builder: Use Centos 7 cloud image symlink https://review.openstack.org/183185 | 19:08 |
| openstackgerrit | Merged openstack/diskimage-builder: doc: small snippet about operating system elements https://review.openstack.org/183616 | 19:09 |
| *** dguerri`away is now known as dguerri | 19:15 | |
| *** gfidente|afk is now known as gfidente | 19:18 | |
| *** david-lyle has joined #tripleo | 19:18 | |
| *** panda has quit IRC | 19:21 | |
| *** panda has joined #tripleo | 19:21 | |
| *** cwolferh has quit IRC | 19:22 | |
| *** morazi has quit IRC | 19:25 | |
| *** victor_lowther has quit IRC | 19:25 | |
| *** victor_lowther has joined #tripleo | 19:25 | |
| *** julim has quit IRC | 19:25 | |
| *** david-lyle has quit IRC | 19:27 | |
| *** eghobo has quit IRC | 19:31 | |
| *** weshay has quit IRC | 19:35 | |
| *** cody-somerville has quit IRC | 19:45 | |
| *** mcornea has joined #tripleo | 19:48 | |
| *** mcornea has quit IRC | 19:48 | |
| *** eghobo has joined #tripleo | 20:28 | |
| *** rlandy has joined #tripleo | 20:38 | |
| *** Marga_ has joined #tripleo | 20:42 | |
| *** david-lyle has joined #tripleo | 20:44 | |
| *** Marga_ has quit IRC | 20:45 | |
| *** Marga_ has joined #tripleo | 20:45 | |
| *** Marga_ has quit IRC | 20:45 | |
| *** Marga_ has joined #tripleo | 20:46 | |
| *** gfidente has quit IRC | 20:52 | |
| *** jayg is now known as jayg|g0n3 | 20:56 | |
| *** untriaged-bot has joined #tripleo | 21:00 | |
| untriaged-bot | No untriaged bugs so far! \o/ | 21:00 |
| *** untriaged-bot has quit IRC | 21:00 | |
| *** weshay has joined #tripleo | 21:00 | |
| *** eghobo has left #tripleo | 21:10 | |
| *** lblanchard has quit IRC | 21:15 | |
| *** rlandy has quit IRC | 21:16 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Rename $name variable to $module_name https://review.openstack.org/180499 | 21:17 |
| *** trown is now known as trown|outttypeww | 21:18 | |
| openstackgerrit | Merged openstack/tripleo-puppet-elements: Define $DIB_INSTALLTYPE_puppet_modules https://review.openstack.org/179101 | 21:20 |
| *** weshay has quit IRC | 21:24 | |
| *** david-lyle has quit IRC | 21:24 | |
| *** Marga_ has quit IRC | 21:27 | |
| *** cody-somerville has joined #tripleo | 21:39 | |
| *** weshay has joined #tripleo | 21:55 | |
| *** daneyon has joined #tripleo | 22:11 | |
| *** chlong has quit IRC | 22:12 | |
| *** daneyon has quit IRC | 22:14 | |
| *** weshay has quit IRC | 22:14 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Update delorean URL https://review.openstack.org/184640 | 22:21 |
| *** cody-somerville has quit IRC | 22:32 | |
| *** Marga_ has joined #tripleo | 22:34 | |
| *** cwolferh has joined #tripleo | 22:39 | |
| *** weshay has joined #tripleo | 22:44 | |
| *** david-lyle has joined #tripleo | 22:45 | |
| *** daneyon has joined #tripleo | 22:45 | |
| *** Marga_ has quit IRC | 22:48 | |
| *** pelix has quit IRC | 22:49 | |
| *** daneyon has quit IRC | 22:52 | |
| *** daneyon has joined #tripleo | 23:03 | |
| *** daneyon has quit IRC | 23:11 | |
| *** david-lyle has quit IRC | 23:13 | |
| *** athomas has joined #tripleo | 23:15 | |
| *** Marga_ has joined #tripleo | 23:18 | |
| *** panda has quit IRC | 23:21 | |
| *** panda has joined #tripleo | 23:22 | |
| *** shakamunyi has joined #tripleo | 23:27 | |
| *** Marga_ has quit IRC | 23:30 | |
| *** chlong has joined #tripleo | 23:34 | |
| *** shakamunyi has quit IRC | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!