Monday, 2015-05-18

*** chlong has joined #tripleo00:09
*** Guest11507 has joined #tripleo00:28
*** Guest11507 has quit IRC00:35
*** julim has joined #tripleo00:50
*** sdake has quit IRC01:06
*** sdake has joined #tripleo01:07
*** julim has quit IRC01:15
*** julim has joined #tripleo01:32
*** julim has quit IRC01:44
*** sdake has quit IRC02:00
openstackgerritYanis Guenane proposed openstack/tripleo-heat-templates: Update to reflect puppet-pacemaker changes  https://review.openstack.org/18310302:34
openstackgerritYanis Guenane proposed openstack/tripleo-heat-templates: Fix RabbitMQ startup race  https://review.openstack.org/18139802:34
*** chlong has quit IRC02:58
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145517503:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985203:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985403:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480303:00
openstackLaunchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107)03:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/145275203:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480203:00
*** untriaged-bot has quit IRC03:00
openstackLaunchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)03:00
openstackLaunchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)03:00
openstackLaunchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New]03:00
openstackLaunchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed]03:00
openstackLaunchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New]03:01
*** tserong has quit IRC03:06
*** chlong has joined #tripleo03:10
*** tserong has joined #tripleo03:24
*** tserong has joined #tripleo03:24
*** lazy_prince has joined #tripleo04:04
*** daneyon has quit IRC04:16
*** chlong has quit IRC04:18
*** chlong has joined #tripleo04:35
*** noslzzp has joined #tripleo04:39
*** julim has joined #tripleo04:48
*** julim has quit IRC04:49
*** julim has joined #tripleo04:50
*** aufi has joined #tripleo05:01
*** masco has joined #tripleo05:04
*** noslzzp has quit IRC05:13
*** aufi has quit IRC05:19
*** jprovazn has joined #tripleo05:58
*** yog__ has joined #tripleo06:16
*** julim has quit IRC06:37
*** ukalifon1 has joined #tripleo06:37
*** sdake has joined #tripleo06:40
*** julim has joined #tripleo06:40
*** mmagr has joined #tripleo06:43
*** sdake has quit IRC06:45
*** yamahata has joined #tripleo06:59
*** al has quit IRC07:03
*** al has joined #tripleo07:04
*** jtomasek has joined #tripleo07:15
*** cody-somerville has joined #tripleo07:16
*** Goneri has joined #tripleo07:32
*** jistr has joined #tripleo07:34
*** yog__ has quit IRC07:39
*** ifarkas has joined #tripleo07:40
*** chlong has quit IRC07:40
openstackgerritMerged openstack/tripleo-incubator: Add support for overriding cache just for seed  https://review.openstack.org/16310807:41
*** yamahata has quit IRC07:43
*** yog__ has joined #tripleo07:44
*** ishant has joined #tripleo07:52
*** julim has quit IRC07:56
*** masco_ has joined #tripleo08:04
*** masco has quit IRC08:07
*** mcornea has joined #tripleo08:07
*** cody-somerville has quit IRC08:13
*** pblaho has joined #tripleo08:14
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Set up corosync using hostnames rather than IPs  https://review.openstack.org/17773508:21
mariosjistr: rebased ^^^ seems node_names landed already somewhere else? (rebased, please check)08:21
jistrmandre: thx! looks good08:22
mariosIf0c37a86fa8b5aa6d452129bccf7341a3a3ba66708:22
mariosjistr: https://review.openstack.org/#/c/177765/08:23
mariosjistr: sorry, bug was auto reassigned to me08:27
mariosjistr: are you jiri-stransky08:28
mariosjistr: or jistran08:28
jistrjistr: i'm jistr on launchpad too, i'll take it :)08:28
mariosjistr: yeah thanks08:28
*** gfidente has joined #tripleo08:34
gfidentemorning :)08:35
jistrmorning :)08:36
jistrgfidente: seen this? https://review.openstack.org/#/c/183934/   i'm thinking now if we can actually solve this rather than remove it from pcmk08:38
gfidenteah so it actually applies to the other services08:40
jistryes...08:40
*** crinkle has quit IRC08:40
gfidenteso I was thiking08:41
jistri'm adding one more step08:42
jistrto the beginning08:42
jistrto perform cluster setup08:42
gfidentesec, I was thinking to08:42
gfidentemove the pacemaker steps08:43
gfidenteso let's have a call08:43
gfidenteokay?08:43
gfidenteso the steps I see are08:44
gfidente1. setup haproxy/rabbit/galera (probably others, mongo, redis)08:45
gfidente2. configure pcmk, bring up VIPs and add infra services to pcmk08:46
gfidente3. db sync08:46
gfidente(and setup of all openstack services)08:46
gfidente4. add openstack services to pcmk08:46
gfidentehow do they look?08:47
gfidenteand I would proceed so, I would remove from your update the deps on rabbitmq and the temorary haproxy removal08:51
gfidenteand make a single update, meant to fix the steps, on top of your08:51
gfidenteAND08:52
gfidenteif you're okay with it,08:52
gfidenteI'll do it now :)08:52
*** rlandy has joined #tripleo08:53
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145517509:00
openstackLaunchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107)09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145604709:00
openstackLaunchpad bug 1456047 in tripleo "[t-h-t] Pacemaker orchestration issue" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985209:00
openstackLaunchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)09:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985409:00
openstackLaunchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480309:00
openstackLaunchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480209:00
openstackLaunchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New]09:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/145275209:00
openstackLaunchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed]09:00
*** untriaged-bot has quit IRC09:00
gfidentejistr ?09:01
jistrgfidente: sorry i'm getting pinged re something else09:01
jistri'll be back on this09:01
gfidenteack I will try the change09:01
jprovazngfidente: hi09:06
gfidentejprovazn, morning :09:06
dasmo/09:06
jprovazngfidente: HA puppet question... I wonder if there are any plans for future to deal with scaling down on puppet modules level? e.g. if a compute nodes is removed during stack update, then nova puppet module would automatically unregister the node? (not saying puppet should support this, just wonder if anything iike this would be planned potentially)09:08
gfidentejprovazn, to be honest, looks complex issue to me09:15
gfidentenodes don't have any understanding of the deployment state09:15
gfidenteas it was with elements09:16
gfidenteso even if we were to trigger some action before the actual removal of the server resource09:16
gfidenteI wouldn't be sure how to tell "remove this node from config list"09:18
gfidentethere is people more expert than me with puppet though who might have approached this before09:18
gfidentethinking of jistr, jayg|g0n` or cwolferh09:18
jprovazngfidente: yes, it's complex, on the other side we already support scaleup of services, scaledown would be related (related!=similar :)09:19
jprovazngfidente: not sure what you mean by "how to tell "remove this node from config list"" - this instruction (and which node) comes from heat09:20
gfidenteyeah sorry09:20
jistryeah we haven't looked into scaling down yet09:20
gfidenteso I meant puppet works by 'applying a config' like orc did09:21
jprovaznjistr: yes, sure - I'm not even saying you should, I just wonder if this is something you guys would have potentially on your TODO list?09:21
gfidenteassuming you could, from heat, provide a new structuredconfig with some instructions of which node should be removed09:21
gfidenteI don't think puppet could use that alone and turn it into a 'remove node from cluster' action09:22
gfidenteand it doesn't look like re-applying the config with one node less is what we need in this scenario, so the easiest path doesn't seem to be able to cope with what we want09:22
jprovazngfidente: yes, it would require extra logic in puppet module to e.g. detect that nodes set decreased, get list of removed ndoes and call some custom actions to de-register them (e.g. from rabbitmq cluster)09:24
gfidenteyeah have some understanding of state09:25
jprovaznok, thanks, lunch09:25
gfidenteso it's not on my TODO list09:26
*** links has joined #tripleo09:26
gfidenteand it doesn't look it would be quick to do :)09:26
jistrone of the typical puppet problems is that puppet gets only fed the target configuration, not the previous one to compare with09:26
gfidenteif it's on other people TODO I don't know though09:27
jistrso removing things via puppet is traditionally not very straightforward09:27
gfidenteyeah09:27
jistrgfidente: so you'd flip the order of setting up the "support services" vs. pcmk?09:33
gfidentenot only flipping to be hones09:35
gfidenteif we want to use steps09:35
gfidentewe need at least 409:35
gfidente1 infra setup, 2 infra pcmk, 3 openstack setup, 4 openstack pcmk09:36
gfidenteI have an idea09:37
*** pelix has joined #tripleo09:38
jistrcould be 1 all setup, 2 infra pcmk, 3 openstack pcmk?09:38
gfidenteabout disabling all services first so we don't need to repeat step 3/4 for db_sync09:38
gfidentejistr, yeah we need something in between 2/3 for db_sync09:39
jistryeah09:39
gfidentebut sure we could do the config all in one ste09:39
jistrso sounds like:09:41
gfidentelet me try something cause it is easier to do than to talk09:41
gfidenteit's just copy/paste :P09:41
gfidentebut let's agree first on what the final result should be09:42
*** pblaho is now known as pblaho__09:42
gfidentesounds like?09:43
jistrsounds like 1 all setup, 2 infra pcmk, 3 dbsync, 4 openstack pcmk09:44
gfidenteworks for me09:44
gfidenteis there a reason to use step3 only for db_sync though compared to setting up the openstack services there?09:45
jistrnot really, but seems to me like having all config written as soon as possible is not a bad approach either09:50
gfidenteack09:50
jistralso looking into the code, maybe db sync can stay in step 209:50
jistrit just needs a dep on galera09:50
jistrwhich it already has09:50
jistri'm also trying this in parallel09:50
jistri already added the initial step though09:50
jistrso i might need to remove it again09:51
*** jistr has quit IRC09:53
*** jistr has joined #tripleo09:55
*** pblaho has joined #tripleo09:58
*** pblaho has quit IRC09:59
*** pblaho has joined #tripleo09:59
*** pblaho__ has quit IRC10:01
*** mmagr has quit IRC10:02
*** mmagr has joined #tripleo10:03
*** jistr_ has joined #tripleo10:06
*** jistr has quit IRC10:07
*** mmagr has quit IRC10:11
*** pblaho has quit IRC10:11
*** mmagr has joined #tripleo10:11
*** pblaho has joined #tripleo10:12
*** regebro has joined #tripleo10:12
*** masco_ has quit IRC10:28
*** masco_ has joined #tripleo10:33
*** pblaho has quit IRC10:37
*** pblaho has joined #tripleo10:38
*** crinkle has joined #tripleo10:40
*** jistr_ is now known as jistr10:44
*** mmagr has quit IRC10:47
*** mmagr has joined #tripleo10:48
*** thrash|wknd is now known as thrash11:13
*** dguerri_ has joined #tripleo11:22
*** panda|gone is now known as panda11:22
*** pblaho has quit IRC11:24
*** mmagr is now known as mmagr|afk11:25
*** jistr has quit IRC11:34
* gfidente figured has to redo the whole galera thing11:35
*** jistr has joined #tripleo11:35
*** pblaho has joined #tripleo11:36
*** dguerri_ is now known as dguerri11:37
*** dguerri is now known as dguerri_11:37
gfidentejistr, I am done11:41
gfidentebut I found weird stuff with the galera scripts11:41
gfidenteI'll show you from code11:41
jistrgfidente: cool :) i was just debugging oslo_db.exception.DBConnectionError: (OperationalError) (2003, 'Can\'t connect to MySQL server on \'192.0.2.3\' (111 "Connection refused")')11:42
jistrnot sure if that's related11:42
gfidenteI don't think it is11:42
*** dguerri_ is now known as dguerri11:42
jistrgfidente: ok nevermind, let's discuss what you've found11:43
*** barra204 has quit IRC11:53
*** shakamunyi has quit IRC11:54
*** ishant has quit IRC12:04
gfidentejistr, any idea how do I unlock https://review.openstack.org/#/c/180229/ ?12:12
jistrgfidente: i've been just looking at it lol12:13
jistrgfidente: i think reverify is correct12:13
jistrgfidente: it might be that only cores can do that, i dunno12:13
jistrit doesn't come up very often12:13
jistrso we'll need to get hold of someone from puppet-core to reverify i think12:13
*** mcornea has quit IRC12:14
*** mcornea has joined #tripleo12:14
*** mcornea has joined #tripleo12:16
jistrgfidente: i see it in the gate queue in fact, so it might be that all is well and it will be merged in a while12:17
*** mcornea has quit IRC12:18
*** mcornea has joined #tripleo12:18
jistrgfidente: it's in ;)12:21
gfidentefiuuu12:21
*** masco_ has quit IRC12:22
gfidentejistr, so I can't move the openstack resources into step 1 today12:23
gfidentebecause they are not yet started by pacemaker12:23
gfidenteand they need the infra services to work12:24
gfidenteI think this can be done later12:24
jistrgfidente: yeah you're right, i've hit that too a while ago. I tried to do the minimal set of changes to get us green, still have troubles on db syncs though12:24
gfidentefor each and every service, unless we do everything now12:24
gfidenteyeah it won't be easy12:24
gfidentegive me a minute to push the changes12:24
jistrgfidente: btw do you get db syncs running on all nodes or just bootstrap?12:24
gfidentejust bootstrap12:24
jistrok... i hit an issue on friday that i've seen db syncs in all os-collect-config logs12:25
jistri mean on all controller nodes12:25
gfidenteoh well12:25
*** trown|outttypeww is now known as trown12:25
gfidenteI meant the purpose of the change is to run it on single node only12:25
gfidenteI think the DSN setup was running db_sync12:26
gfidenteregardless of sync_db status, that's why it was failing previously12:26
*** mestery has joined #tripleo12:27
gfidentejistr, if I wanted to test on top of the update12:33
gfidentecan I just update the puppet-pacemaker module in the image?12:33
*** jayg|g0n` is now known as jayg12:34
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure HAProxy, Galera and MongoDB before start  https://review.openstack.org/18404312:35
jistrgfidente: yeah i'd say so12:35
jistrbtw ^^12:35
jistrthat got to a successful stack-create12:35
gfidentesec12:35
jistrin the end it looks like we don't have to do that much shuffling to get things green... ~60 lines12:37
gfidenteI did lots of changes!12:38
gfidente:{12:38
gfidentebut for the good, give me a min12:38
gfidenteI want to test it as well before pushing12:38
jistrheh12:40
jistr+ wait_for -w 300 --delay 10 -- nova service-list --binary nova-compute '2>/dev/null' '|' grep 'enabled.*\ up\ '12:40
jistrTiming out after 300 seconds:12:40
jistrCOMMAND=nova service-list --binary nova-compute 2>/dev/null | grep enabled.*\ up\12:40
jistrcontrollers are fine, but the compute node didn't get registered with controllers12:41
jistrnova-compute log is filled with this12:48
jistr2015-05-18 12:46:41.466 4085 WARNING nova.conductor.api [req-f72df90b-8e66-4e8b-8ec0-ba6362b83e61 - - - - -] Timed out waiting for nova-conductor.  Is it running? Or did this service start before nova-conductor?  Reattempting establishment of nova-conductor connection...12:48
jistrotherwise all seems well, it even connected to rabbit on the VIP successfully12:48
jistrso... WUT12:48
gfidentethe timeout message is fine12:49
gfidentecomputes start before the controllers12:50
gfidenteit is supposed to recover though when conductor comes up12:50
jistrmarios, gfidente: yeah but it didn't recover :( So, that's funny, i just needed to restart the openstack-nova-compute service on the compute node, and now it's registered. So it only recovered after restarting nova-compute. Does that ring a bell to someone? I vaguely recall something like this but don't recall the cause.12:51
*** mmagr|afk is now known as mmagr12:51
*** dguerri is now known as dguerri_12:52
gfidenteso did you get what in the compute logs before the restart?12:52
gfidentestill same timeout message or exceptions of some kind related to message transport?12:53
jaygjistr: I have seen that before too12:53
jaygnot in tripleo, just regular other deployments12:53
jaygI think compute is trying to wait on a response to a message id from rabbit that api doesnt have12:53
jistrgfidente: no exceptions, connected to AMQP fine, just repeated that " Timed out waiting for nova-conductor." message i pasted above, even though at that time 3 instances of conductor were up and running and registered12:54
jistrjayg: thanks, that sounds like it. Did you get to solve it anyhow in Astapor?12:55
jaygjistr: no, we recommend starting control nodes first, otherwise you need a service restart on compute12:56
* jayg has this idea that perhaps a redelivery setting is needed in rabbit queues, but that is just an idea12:57
jistrhmm yeah... the "Or did this service start before nova-conductor?" part of what i pasted above sounds like a hint too :) So we may need to hook compute nodes to get configured after controllers are done too...12:59
jistrgfidente: ^12:59
gfidentejistr, so to do that we can add a depends_on13:00
jistryeah13:00
gfidenteI am unsure though if we should try to fix the issue13:00
gfidenteor workaround it13:00
gfidentemaybe it's better to workaround for now13:00
jaygjistr: yeah, nova clearly doesn't expect them to start first, but it does seem like it should be able to recover, which is more what my idea was directed at13:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Update to reflect puppet-pacemaker changes  https://review.openstack.org/18310313:01
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Test resources reorganization  https://review.openstack.org/18404713:01
gfidenteauch13:01
gfidentedidn't want to refresh the update, but did the checkout before your last update!13:01
gfidenteanyway, the test, there are a bunch of not strictly related changes13:02
jaygguys, is there now anything needed to get an HA deploy besides OS::TripleO::ControllerConfig, EnablePacemaker and setting number of controllers?13:03
gfidentebut I would like to port stuff from there into a number of small changes13:03
jaygI had all 3 set friday nd was not getting any pacemaker anything deployed13:03
gfidentejistr, possibly on top of your haproxy/galera/mongodb13:03
*** regebro has quit IRC13:04
jistrgfidente: ack, i see you removed the dep on the RabbitMQ fix, i'll add that one to the commit on top instead, since it has the same nature as the  haproxy/galera/mongodb fix13:05
gfidentejistr, sec13:05
gfidentejistr, that is only a test I will -113:05
gfidentewhat I would is, checkout your change here: https://review.openstack.org/18404313:05
gfidenteand add changes from the test submission on top of that one13:06
gfidenteif you wanted to move rabbitmq into the galera/haproxy/mongodb13:06
gfidentethan it is pretty much what the test submission does13:06
jistrgfidente: i see you have a bigger diff there though, incl. the db syncs13:07
jistrmaybe we should split it at least a bit13:07
gfidentewe definitely should!13:07
gfidentethat's why I called test13:07
gfidenteI want to abandon it13:07
gfidentebut that is a working version13:08
jistrok, but it also updated the one below it, which we'll have to restore in some way13:08
jistreither by re-adding the depencency on the rabbitmq fix13:08
gfidenteyeah that is my mistake13:08
jistror by moving the rabbitmq fix to the galera/haproxy/mongo patch13:08
gfidenteI did checkout of your change before you updated13:09
gfidenteso I had a different copy13:09
*** AzherKhan has joined #tripleo13:12
gfidentejistr, no worries for now, ignore the test for now13:13
gfidentejistr, I am building on top so we don't redo the work13:13
gfidenteand it's easier to review13:14
jistrgfidente: on top of https://review.openstack.org/184043 as it is now?13:14
gfidenteexactly yes13:14
jistrgfidente: ack13:15
jistrthanks13:15
*** sdake has joined #tripleo13:19
*** AzherKhan has quit IRC13:20
*** dguerri_ is now known as dguerri13:20
*** sdake has quit IRC13:24
*** AzherKhan has joined #tripleo13:30
*** yog__ has quit IRC13:31
*** athomas has joined #tripleo13:31
*** mestery has quit IRC13:33
*** athomas_ has joined #tripleo13:33
*** Khaazher has joined #tripleo13:35
*** athomas has quit IRC13:37
spredzy|afkHello all13:37
*** spredzy|afk is now known as spredzy13:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove unused enable_pacemaker setting from templates  https://review.openstack.org/18405713:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Rename Puppet post config steps name to match the actual step  https://review.openstack.org/18405813:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Ensure sync_db is consumed by Keystone and Cinder classes  https://review.openstack.org/18405913:38
gfidentespredzy ! ^_^13:38
gfidentehow's going?13:38
spredzyahah good, still a lil bit jetlag :)13:39
spredzyI spent part of the week end debugging the tripleo HA job until I realized this https://bugs.launchpad.net/tripleo/+bug/145604713:39
openstackLaunchpad bug 1456047 in tripleo "[t-h-t] Pacemaker orchestration issue" [Undecided,New]13:39
*** AzherKhan has quit IRC13:39
spredzybut it seems that you guys are working on it already :p13:39
gfidentespredzy, yeah on different angles13:39
spredzygfidente, yeah but ti will address the same issue so at the end it should work just fine, no more race issue with the setup :)13:40
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Let sync_db param alone orchestrate on which node db_sync run  https://review.openstack.org/18406313:44
*** sdake has joined #tripleo13:45
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure computes after controllers  https://review.openstack.org/18406413:45
*** mmagr is now known as mmagr|afk13:45
jistrgfidente: unfortunately, locally this doesn't solve the issue for me... ^13:46
*** athomas_ has quit IRC13:46
jistri wonder what's the scoop then if not ordering13:46
jistrmorning spredzy13:46
gfidentespredzy, GO TRIPLEO!13:47
spredzyjistr, hey :)13:47
spredzyahaha13:47
spredzyjistr, sorry I prob. mess with some of your reviews this week end, hopefully I put back everything to normal :)13:48
jistrspredzy: haha no prob i didn't notice anything wrong so i think all is well :)13:49
spredzyjistr, cool. So I had a question about pacemaker and astapor. How the orchestration takes place ? How  did it ensure configuration of services was done on *all* nodes before declaring the actual pacemaker resource ?13:50
jistrspredzy: a trick with pacemaker properties. Basically pacemaker has a built in a simple key/value store where k/v pairs can be assigned to nodes. So after config is written on a node, it sets a property on itself, and when all the nodes have this property, then some further action (e.g. creating the pacemaker resource) can be taken. (hopefully i didn't mess this up, jayg probably knows this better)13:53
*** mmagr|afk is now known as mmagr13:53
*** zb has joined #tripleo13:53
jistrspredzy: this allows to deploy all controllers in a single puppet run. Instead of Heat steps they use Pacemaker properties to perform inter-node orchestration.13:54
*** mmagr is now known as mmagr|afk13:54
jaygjistr: the mechanism (properties) is part of pacemaker, but writing to them and making them into usable facts for future runs was part of quickstack13:54
*** Khaazher has quit IRC13:55
spredzyjistr, [...] it sets a property on itself [...] via puppet ?13:56
jistrspredzy: probably code will say it better https://github.com/redhat-openstack/astapor/blob/master/puppet/modules/quickstack/manifests/pacemaker/glance.pp#L149-L16813:57
jistrspredzy: so yeah, it sets a pacemaker property on itself via puppet exec, then waits till all nodes have that property, then it creates the resources13:58
spredzyjistr, ack I see thanks13:58
*** lblanchard has joined #tripleo14:01
*** zbitter has joined #tripleo14:04
*** yamahata has joined #tripleo14:04
*** spredzy is now known as spredzy|afk14:04
*** mcornea has quit IRC14:04
*** mcornea has joined #tripleo14:05
*** zb has quit IRC14:07
gfidentejistr, how do they look changes up to that point?14:10
jistrgfidente: sorry could you please rephrase, i'm not sure what you're referring to exactly14:13
gfidentehow do they look up to: https://review.openstack.org/#/c/184063/14:14
gfidenteI am going to do the actual steps reorganization on top of that14:14
*** julim has joined #tripleo14:15
*** dguerri is now known as dguerri_14:16
*** zb has joined #tripleo14:17
*** dguerri_ is now known as dguerri14:18
*** jprovazn has quit IRC14:18
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure computes after controllers  https://review.openstack.org/18406414:20
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Update to reflect puppet-pacemaker changes  https://review.openstack.org/18310314:20
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure HAProxy, Galera and MongoDB before start  https://review.openstack.org/18404314:20
*** zbitter has quit IRC14:21
*** athomas has joined #tripleo14:21
jistrgfidente: i updated https://review.openstack.org/#/c/183103/14 to put back the dependency on the rabbitmq fix. I'll look at the follow up fixes now, sorry, i got pinged because of something else before.14:22
gfidentethat is fine14:22
gfidenteI will rebase eventually14:22
gfidentefor now I am pushing changes14:22
gfidenteuntil I get to the steps separation, then we update/rebase as needed14:23
*** zbitter has joined #tripleo14:23
*** zaneb has joined #tripleo14:26
*** zb has quit IRC14:27
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Enable VIPs and HAProxy from Pacemaker in step 2  https://review.openstack.org/18407214:29
*** zbitter has quit IRC14:30
*** zb has joined #tripleo14:32
gfidentejistr, good to add shardy on the review14:33
gfidenteI wanted to make the names consistent with the actual step14:34
*** zbitter has joined #tripleo14:34
gfidenteso hopefully validation is not basing on the config name14:34
gfidentebut if it is, we can squash that change14:34
mariosgfidente: am looking at that now. any context why they were named like that?14:34
gfidentehi marios14:34
marioso/14:34
marios:)14:35
gfidenteI don't think there is a reason, it wouldn't be nice to be forced to give some name to a structuredconfig14:35
openstackgerritDmitry Teselkin proposed openstack/diskimage-builder: Allow dib to be lauched from venv  https://review.openstack.org/16800214:35
*** zaneb has quit IRC14:36
*** zb has quit IRC14:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Enable VIPs and HAProxy from Pacemaker in step 2  https://review.openstack.org/18407214:39
*** noslzzp has joined #tripleo14:39
jistrgfidente: yeah i don't think it would prevent you from setting a breakpoint there (i guess you can stop on arbitrary resources) but i still wonder if we shouldn't keep some conventions... without having the "Step<N>" names consecutive, it's not obvious where ringbuilder.pp happens and that it might be a spot where the breakpoint might be useful14:39
gfidentejistr, indeed so let's see what shardy suggests14:40
jistrbut as i said, i don't feel strong about it. Syncing the names with the "step: N" variable has value as well14:40
gfidentethe change itself is not really needed for the restructuring, we can remove it from deps if we want14:40
jistryeah if we could treat it as a separate thing that would be cool14:40
mariosgfidente: +1 to trimming/prioritizing reviews14:41
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure computes after controllers  https://review.openstack.org/18406414:43
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure HAProxy, Galera and MongoDB before start  https://review.openstack.org/18404314:43
jistrgfidente: just a small fix, hopefully that won't break a rebase for you too much https://review.openstack.org/#/c/184043/2..3/puppet/manifests/overcloud_controller_pacemaker.pp14:44
gfidentejistr, I will rebase, no worries14:44
gfidentejistr, it's more about checking the changes together so we agree on where we are heading14:45
*** dguerri is now known as dguerri_14:45
*** lazy_prince has quit IRC14:47
*** dguerri_ is now known as dguerri14:50
*** zb has joined #tripleo14:51
*** zaneb has joined #tripleo14:54
*** zbitter has quit IRC14:55
*** zb has quit IRC14:57
*** yamahata has quit IRC14:58
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Consolidate use of $pacemaker_master in step 2  https://review.openstack.org/18407814:59
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145517515:00
openstackLaunchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107)15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145604715:00
openstackLaunchpad bug 1456047 in tripleo "[t-h-t] Pacemaker orchestration issue" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985215:00
openstackLaunchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)15:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985415:00
openstackLaunchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480315:00
openstackLaunchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480215:00
openstackLaunchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/145275215:00
openstackLaunchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed]15:00
*** untriaged-bot has quit IRC15:01
*** MasterPiece has joined #tripleo15:02
gfidentejistr, shardy didn't know that depended on the resource name15:04
*** links has quit IRC15:05
*** masco has joined #tripleo15:05
gfidentejistr, btw, before I rebase/squash can you check the submissions up to https://review.openstack.org/#/c/184078/ and see if they look valid to you?15:05
gfidentejistr, not necessarily review but make sure they go into right direction so I can update/rebase and see who wants to merge :)15:05
jistrgfidente: i think this one won't fly... https://review.openstack.org/#/c/184063/ -- did you try to run with it yet? i think db creation != db sync15:06
gfidenteunless I tried with some checkout before removing it15:07
jistrgfidente: i think we need to create the databases there first, and then the services can perform their db syncs. I think the services themselves don't have permissions to create DBs for them.15:07
jistrhmm interesting... i could be mistaken of course, but it would be surprising to me how can such a thing work then... Astapor has to do this too afaik https://github.com/redhat-openstack/astapor/blob/b31bfd25ab07ab1ba2941b7d041c176f59f7a8f1/puppet/modules/quickstack/manifests/db/mysql.pp15:08
*** cody-somerville has joined #tripleo15:12
*** ifarkas has quit IRC15:12
*** zb has joined #tripleo15:14
jaygjistr: we actually don't use db/mysql anymore, but do use galera/db, which does something similar15:17
*** Marga_ has joined #tripleo15:17
*** Marga_ has quit IRC15:18
*** zaneb has quit IRC15:18
*** Marga_ has joined #tripleo15:18
*** zbitter has joined #tripleo15:20
gfidenteokay so I am updating and fixing where neede :)15:21
*** cwolferh has quit IRC15:22
*** zb has quit IRC15:23
*** daneyon has joined #tripleo15:24
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove unused enable_pacemaker setting from templates  https://review.openstack.org/18405715:24
*** julim has quit IRC15:30
*** zb has joined #tripleo15:32
jaygjistr: to recap what I was trying to say on call -  is there now anything needed to get an HA deploy besides OS::TripleO::ControllerConfig, EnablePacemaker and setting number of controllers?15:32
jaygthough it looks like EnablePacemaker one is ignored with upcoming patch...15:33
gfidenteshardy, is this breaking validation https://review.openstack.org/#/c/18405815:34
gfidenteor is it just skipping one of the steps15:34
gfidenteand running once after both ringbuilder and the subsequent step are executed?15:34
jistrjayg: i think EnablePacemaker is ignored already...15:34
gfidentejayg, jistr call means puppeteers?15:35
jistrjayg: so what is the problem you are hitting? I think we can start debugging with: Do the nodes get created? Does Puppet run on the nodes at all?15:35
jistrgfidente: yeat15:35
jistr*yeah15:35
jaygyes, puppet runs, but no pacemaker config happens at all15:36
gfidentejayg, which change did you checkout on top of master?15:36
jaygso when I tun pcs status at end, I get:15:36
*** zbitter has quit IRC15:36
jaygrror: cluster is not currently running on this node15:36
jistrjayg: also no sign of pacemaker in 'journalctl -u os-collect-config'?15:36
jistr(to see whether the pacemaker config wasn't even attempted)15:37
*** julim has joined #tripleo15:37
gfidentejayg, I think you just miss the overcloud_custom_env file, but I am interested into which changes are you testing?15:37
jayggfidente: I had your rabbitmq startup race patch, and jistr puppet-pacemaker compat patch15:38
jayggfidente: so you mean, for example, pacemaker.yaml we set up on this machine together?15:39
jaygOVERCLOUD_CUSTOM_HEAT_ENV=/home/tripleo/pacemaker.yaml15:39
gfidenteexactly, in there you give15:39
gfidenteresource_registry:15:39
gfidente  OS::TripleO::ControllerConfig: tripleo/tripleo-heat-templates/puppet/controller-config-pacemaker.yaml15:39
jaygyeah, since I have that in home, I tried a couple paths, since the relative one didn't seem to 'take'15:40
jaygI started with what you say above though15:40
jaygsame result, no pacemaker15:40
gfidenterelative is the only which should15:40
gfidentewell or absolute15:41
*** zbitter has joined #tripleo15:41
*** masco has quit IRC15:41
jaygyeah, I tried both, same result15:41
gfidentethat is all you should need, assuming you have some updated version of t-h-t15:41
jaygthen I went back to directly altering overcloud-resource-registry-puppet.yaml, still nothing15:41
*** ukalifon1 has quit IRC15:42
jaygthis is why I wondered if perhaps there were some new setting I missed, this at least set up pacemaker before, even though I had other problems still15:42
*** zb has quit IRC15:45
jistrjayg: do you have the var actually exported, or just set? can be tested by running: bash -c 'echo $OVERCLOUD_CUSTOM_HEAT_ENV'15:45
jaygyep, it is set - /home/tripleo/pacemaker.yaml15:46
jaygwhy the extra bash -c on that?15:46
jistrbecause if15:46
jistrecho $OVERCLOUD_CUSTOM_HEAT_ENV15:46
jistrprints the path15:46
jistrbut this:15:47
jistrbash -c 'echo $OVERCLOUD_CUSTOM_HEAT_ENV'15:47
jistrdoesn't print the path, then that's the problem15:47
jaygah, k - they both print the same thing15:47
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Move NTP and Memacache respectively into step 1 and step 2  https://review.openstack.org/18406315:47
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Ensure sync_db is consumed by Keystone and Cinder classes  https://review.openstack.org/18405915:47
jistrack, hmm15:47
*** Marga_ has quit IRC15:49
*** athomas has quit IRC15:50
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Enable VIPs and HAProxy via Pacemaker from step 2  https://review.openstack.org/18407215:50
*** zbitter has quit IRC15:52
*** julim has quit IRC15:52
*** dasm is now known as dasm|afk15:59
*** akrivoka has joined #tripleo16:00
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Consolidate use of $pacemaker_master in step 2  https://review.openstack.org/18407816:07
*** cody-somerville has quit IRC16:13
gfidentemarios, https://review.openstack.org/#/c/177722/316:13
*** jistr has quit IRC16:17
gfidentejtomasek, found anything about the updates to OPM?16:21
*** penick has joined #tripleo16:22
mariosgfidente: just caught me16:23
*** penick has quit IRC16:23
marioso/ see ya tomorrow have fun summitters16:24
gfidente+a!16:24
gfidente:)16:24
*** cwolferh has joined #tripleo16:27
*** mcornea has quit IRC16:29
*** daneyon has quit IRC16:29
*** pblaho has quit IRC16:29
*** rwsu has joined #tripleo16:35
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Consolidate use of $pacemaker_master in step 2  https://review.openstack.org/18407816:40
*** mmagr|afk is now known as mmagr16:41
*** MasterPiece has quit IRC16:44
*** cody-somerville has joined #tripleo16:52
*** penick has joined #tripleo16:54
*** penick_ has joined #tripleo16:54
*** penick has quit IRC16:58
*** penick_ is now known as penick16:58
*** Goneri has quit IRC17:03
*** akrivoka has quit IRC17:04
openstackgerritMerged openstack-infra/tripleo-ci: Provide a longer timeout for HA scenario  https://review.openstack.org/18356917:13
*** sdake has quit IRC17:26
*** daneyon has joined #tripleo17:26
*** trown is now known as trown|lunch17:26
jtomasekgfidente: the delorean machine is stuck, needs to get back into shape17:32
gfidentehuh and who can do that?17:32
*** daneyon_ has joined #tripleo17:44
*** daneyon has quit IRC17:47
*** noslzzp has quit IRC17:48
*** sdake has joined #tripleo17:53
*** sdake_ has joined #tripleo17:57
*** sdake has quit IRC18:00
*** barra204 has joined #tripleo18:01
*** daneyon_ has quit IRC18:01
*** penick has quit IRC18:08
*** daneyon has joined #tripleo18:10
*** pelix has quit IRC18:13
*** trown|lunch is now known as trown18:18
*** gfidente has quit IRC18:24
*** shakamunyi has joined #tripleo18:25
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add tests for building *-minimal images  https://review.openstack.org/18116218:27
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add smoketest for fedora  https://review.openstack.org/17804018:31
*** radez_g0n3 is now known as radez18:32
*** adam_g has joined #tripleo18:37
*** adam_g has joined #tripleo18:37
*** cody-somerville has quit IRC18:45
*** noslzzp has joined #tripleo18:51
*** MasterPiece has joined #tripleo18:54
*** adrianopetrich has joined #tripleo19:01
*** julim has joined #tripleo19:10
*** panda has quit IRC19:13
*** panda has joined #tripleo19:14
*** shakamunyi has quit IRC19:16
*** barra204 has quit IRC19:17
*** adrianopetrich has quit IRC19:22
*** noslzzp has quit IRC19:27
*** cwolferh has quit IRC19:35
*** sdake_ has quit IRC19:40
*** julim has quit IRC19:42
*** mmagr has quit IRC19:47
*** daneyon has quit IRC19:55
*** radez is now known as radez_g0n320:07
*** shakamunyi has joined #tripleo20:14
*** barra204 has joined #tripleo20:14
*** jayg is now known as jayg|g0n320:24
*** cwolferh has joined #tripleo20:46
openstackgerritMerged openstack-infra/tripleo-ci: Puppet: don't manage /etc/hosts via cloud-init  https://review.openstack.org/17772220:53
openstackgerritMerged openstack-infra/tripleo-ci: Abort failed cherry-pick and reverts  https://review.openstack.org/17394020:58
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145517521:00
openstackLaunchpad bug 1455175 in tripleo "Option to configure gateway through keepalived" [Undecided,New] - Assigned to Mayank (mayank0107)21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145604721:00
openstackgerritMerged openstack/tripleo-incubator: Set DIB_CLOUD_INIT_ETC_HOSTS in Puppet deployments  https://review.openstack.org/17772821:00
openstackLaunchpad bug 1456047 in tripleo "[t-h-t] Pacemaker orchestration issue" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985221:00
openstackLaunchpad bug 1449852 in diskimage-builder "Buidling ramdisk with ironic-agent behind proxy fails" [Undecided,In progress] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)21:00
untriaged-bothttps://bugs.launchpad.net/diskimage-builder/+bug/144985421:00
openstackLaunchpad bug 1449854 in diskimage-builder "Ironic agent ramdisk built using disk-image-create fails with iscsi_ilo driver" [Undecided,Fix committed] - Assigned to Ramakrishnan G (rameshg87) (rameshg87)21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480321:00
openstackLaunchpad bug 1454803 in tripleo "puppet: Neutron is not configured with L2 population" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/145480221:00
openstackLaunchpad bug 1454802 in tripleo "puppet: Neutron does not use Nova notifications" [Undecided,New]21:00
untriaged-bothttps://bugs.launchpad.net/tuskar/+bug/145275221:00
openstackLaunchpad bug 1452752 in tuskar "keystone_authtoken section is wrong in default shipped tuskar.conf.sample" [Undecided,Confirmed]21:00
*** untriaged-bot has quit IRC21:00
*** cody-somerville has joined #tripleo21:02
*** cody-somerville has joined #tripleo21:02
*** lblanchard has quit IRC21:03
*** barra204 has quit IRC21:04
*** shakamunyi has quit IRC21:04
*** barra204 has joined #tripleo21:04
*** shakamunyi has joined #tripleo21:04
*** noslzzp has joined #tripleo21:10
*** cody-somerville has quit IRC21:12
*** dsneddon has joined #tripleo21:18
*** rlandy has quit IRC21:18
*** david-lyle has joined #tripleo21:19
*** BadCub-Traveling is now known as BadCub-At-Summit21:22
*** sdake has joined #tripleo21:22
*** tteggel has quit IRC21:24
*** tteggel has joined #tripleo21:24
openstackgerritDan Prince proposed openstack/tripleo-incubator: Add a puppet disk images YAML file  https://review.openstack.org/18111421:25
*** trown is now known as trown|outttypeww21:28
*** otoolee has quit IRC21:28
*** Goneri has joined #tripleo21:29
*** BadCub-At-Summit is now known as BadCub21:32
*** daneyon has joined #tripleo21:33
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: puppet: use a single image for CI  https://review.openstack.org/17706021:34
*** david-lyle has quit IRC21:36
*** weshay has joined #tripleo21:38
*** sdake has quit IRC21:43
*** dsneddon has quit IRC21:47
*** shakamunyi has quit IRC22:00
*** shakamunyi has joined #tripleo22:00
*** barra204_ has joined #tripleo22:00
*** barra204 has quit IRC22:00
*** barra204_ has quit IRC22:02
*** shakamunyi_ has joined #tripleo22:02
*** barra204_ has joined #tripleo22:02
*** shakamunyi has quit IRC22:02
*** noslzzp has quit IRC22:03
*** athomas has joined #tripleo22:09
*** barra204_ has quit IRC22:10
*** shakamunyi_ has quit IRC22:13
*** Goneri has quit IRC22:15
*** weshay has quit IRC22:19
*** athomas_ has joined #tripleo22:24
*** daneyon has quit IRC22:24
*** athomas has quit IRC22:25
*** athomas_ has quit IRC22:27
*** sdake has joined #tripleo22:48
*** sdake has quit IRC22:57
*** daneyon has joined #tripleo22:58
*** otoolee has joined #tripleo23:01
*** daneyon has quit IRC23:07
*** daneyon has joined #tripleo23:08
*** daneyon has quit IRC23:09
*** chlong has joined #tripleo23:10
*** panda has quit IRC23:13
*** panda has joined #tripleo23:14
*** MasterPiece has quit IRC23:27
*** MasterPiece has joined #tripleo23:30
greghaynesSpamapS: I am having a hard time replicating https://review.openstack.org/#/c/168002 with venv/bin/pip install ./diskimage-builder ... it seems to just work when I do that23:35
greghaynesSpamapS: did you have to do anyting funky to replicate that bug?23:35
*** sdake has joined #tripleo23:41
greghaynesI suspect maybe its been fixed...23:45
*** daneyon has joined #tripleo23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!