Friday, 2015-05-29

greghaynesSpamapS: yea, mirrors suck00:06
greghaynesSpamapS: I am afraid to merge the debian test00:06
greghaynesSpamapS: I wonder if we could convince infra to host a mirror...00:07
*** chuckC_ has quit IRC00:17
*** cody-somerville has quit IRC00:23
*** jeh has joined #tripleo00:36
*** sdake_ has quit IRC00:38
*** sdake has joined #tripleo00:38
*** yamahata has joined #tripleo00:39
*** penick has quit IRC00:45
*** sdake_ has joined #tripleo00:52
*** sdake has quit IRC00:57
openstackgerritIan Wienand proposed openstack/diskimage-builder: Update default Fedora to 22
openstackgerritIan Wienand proposed openstack/diskimage-builder: Work around yum/dnf differences
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add YUM variable to for Fedora >= 22
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup yum downloading
*** cody-somerville has joined #tripleo01:17
*** sdake_ has quit IRC01:17
*** sdake has joined #tripleo01:18
*** GheRivero has joined #tripleo01:22
*** GheRivero has left #tripleo01:23
*** yamahata has quit IRC01:36
*** Slower has quit IRC01:40
*** alop has quit IRC01:43
*** Slower has joined #tripleo01:47
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add virtual IPs for split out networks
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Combine Heat API networks into single net
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Make all-nodes Ip networks configurable
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add virtual IPs for split out networks
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Combine Heat API networks into single net
openstackgerritSteve Baker proposed openstack/os-collect-config: Merge deployments data for collectors heat, request
openstackgerritSteve Baker proposed openstack/os-collect-config: Refactor merging out of cfn collector
*** rwsu has quit IRC02:25
*** sdake_ has joined #tripleo02:40
*** sdake has quit IRC02:43
*** sdake has joined #tripleo02:43
*** sdake_ has quit IRC02:47
*** barra204 has quit IRC03:11
*** barra204 has joined #tripleo03:13
*** thrash is now known as thrash|g0ne03:57
*** tzumainn has quit IRC04:29
*** sdake_ has joined #tripleo04:29
*** sdake has quit IRC04:33
*** lazy_prince has joined #tripleo04:39
*** yog__ has joined #tripleo05:14
*** jeh has quit IRC05:17
*** masco has joined #tripleo05:21
*** ishant has joined #tripleo05:21
*** cody-somerville has quit IRC05:23
openstackgerritIan Wienand proposed openstack/diskimage-builder: Update default Fedora to 22
openstackgerritIan Wienand proposed openstack/diskimage-builder: Work around yum/dnf differences
openstackgerritIan Wienand proposed openstack/diskimage-builder: Cleanup yum downloading
*** AzherKhan has joined #tripleo05:34
openstackgerritIan Wienand proposed openstack/diskimage-builder: Skip backups and other non-relevant files for dib-lint
*** sdake has joined #tripleo05:46
*** sdake_ has quit IRC05:50
dasmgood morning p/06:03
*** cody-somerville has joined #tripleo06:08
*** radek__ has joined #tripleo06:09
*** jtomasek has joined #tripleo06:13
openstackgerritYuriy Zveryanskyy proposed openstack/diskimage-builder: Add Ironic API version to passthru URL for deploy-ironic
*** jprovazn has joined #tripleo06:25
*** bvandenh has quit IRC06:32
*** aufi has joined #tripleo06:36
*** gfidente has joined #tripleo06:40
*** dtantsur|afk is now known as dtantsur06:55
*** ifarkas has joined #tripleo06:59
*** al has quit IRC07:00
*** al has joined #tripleo07:01
*** tzumainn has joined #tripleo07:10
*** yog__ has quit IRC07:12
*** tzumainn has quit IRC07:29
*** paramite has joined #tripleo07:31
*** dtantsur is now known as dtantsur|brb07:35
*** Goneri has joined #tripleo07:35
*** jistr has joined #tripleo07:38
*** yog__ has joined #tripleo07:41
*** pino|work_ has joined #tripleo07:47
*** pino|work_ is now known as pino|work07:47
*** shardy has joined #tripleo07:50
*** sdake_ has joined #tripleo07:55
*** sdake has quit IRC07:59
jistrso one interesting thing, on the *compute* the puppet hook does run but it seems like puppet doesn't even finish compiling the catalog for some reason08:06
gfidenteyeah apparently timing out08:06
gfidentebut yet it is the very same behaviour we get with ps#408:06
gfidentemarios, trying to investigate the nova failures08:07
gfidentemarios, did you try adding the comma on your neutron change?08:08
jistranother interesting thing is, that puppet actually did run there apparently08:09
jistrif i look at nova.conf, e.g. rabbit_hosts is populated08:09
jistrso WUT, how come the puppet run is not logged in os-collect-config logs08:10
jistrhuh maybe the puppet run got stuck, and maybe the puppet hook only prints its output after the puppet run finishes08:11
jistrhmm do we have any "wait for"-like bits in compute manifests08:12
* jistr looks08:12
gfidentein compute logs08:12
gfidenteif you extract the tarball08:12
gfidenteit looks like nova-compute is started but never leaves the08:13
gfidenteTimed out waiting for nova-conductor.  Is it running? Or did this service start before nova-conductor?  Reattempting establishment of nova-conductor connection...08:13
gfidentejistr, did your change to make compute depend on controller ever land?08:13
jistrgfidente: i don't think so, as we've managed to fix our problem without doing that08:14
gfidenteyeah and I think it still applies08:14
gfidenteI wanted to try it out though to see if that helps anyhow08:14
jistri can revive it08:14
gfidenteyeah set a depend on this one so we can see if that passes CI/08:14
gfidenteother than that, controller 2 remains a mistery08:15
jistrdo you remember what was the cause the last time when nova compute couldn't recover?08:17
*** sdake_ has quit IRC08:17
gfidenterabbitmq isolated nodes behind haproxy08:18
jistrah yes08:18
gfidentenot sure why compute wouldn't attach to the isolated conductor running on its same rabbitmq08:19
gfidentebut the config changed quite a bit now08:19
jistryeah that should be ok now08:19
gfidenteor at least, diferent08:19
jistri'm still puzzled by the missing logs from controller3 though08:19
jistrthat shouldn't have anything to do with nova-compute starting too early08:20
gfidenteit doesn't seem to be same issue08:20
gfidenteso to be honest, patchset #4 had same issue on compute, but not on controller 208:21
gfidentewhile previous run of patchset #5 does have same issue on controller 208:21
gfidenteand also same issue with compute node08:22
gfidenteare there other HA jobs which succeeded after memcache/keystone merging?08:23
*** derekh has joined #tripleo08:24
mariosgfidente: no, just poking, good news is i hit it also on my beaker box so trying sthing out in a sec08:25
*** lucasagomes has joined #tripleo08:25
jistrthe last green one is 186416,2 tripleo-heat-templates08:25
*** regebro has joined #tripleo08:27
jistrand this one is green too, has no deps, started 9:33 PM
gfidentebut I also see same error happening08:28
gfidentein unrelated changes08:28
gfidentethat is in relation to compute config timing out08:29
gfidentebut controller 2 does have logs08:29
jistrok i'll just revive the dependency patch then and let CI run on it08:30
*** pblaho has joined #tripleo08:30
*** mcornea has joined #tripleo08:36
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Configure computes after controllers
jistrgfidente: i'm a bit sceptical of that though... i don't think that nova-compute not running correctly would cause the puppet run to hang... I don't think there's any logic in puppet (be it ours or puppet-nova) to say "wait till nova-compute connects to conductor"08:41
mariosthink i got it missing , possibly08:42
jistrbut both problems (puppet time out + nova compute stranded) might have the same cause08:42
hewbroccaHey, who knows whether we are putting ceph MONs on our controllers?08:47
*** regebro has quit IRC08:48
gfidentehewbrocca, yes we are08:50
gfidenteand we can optionally put ceph OSDs on controllers too08:50
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds neutron-server and agents as pacemaker resources
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add metering agent as neutron-server plugin
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add neutron-lbaas as neutron-server plugin & neutron-lbaas-agent
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds neutron-server and agents as pacemaker resources
hewbroccagfidente: cool. Thanks.08:53
jistrgfidente: so i unpacked the logs from one of the controllers, looking at host_info.txt -- only nova-cert and nova-consoleauth are running there. No sign of api/conductor...08:55
jistri thought they'd get killed off because of RAM or something08:57
jistrbut the node has 4GB which should be enough08:57
jistr(i ran into problems locally with 3GB though)08:57
jistrand there are also no logs for nova-api / nova-conductor on the controllers08:57
*** untriaged-bot has joined #tripleo09:00
untriaged-botUntriaged bugs so far:09:00
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora 21 due to Yum multilib error" [Undecided,New]09:00
*** untriaged-bot has quit IRC09:00
gfidentecert is the only one which doesn't have constraints09:01
gfidenteconsoleauth is the root of the constraints tree09:01
jistrah ha, starts to get interesting09:08
mariosguys was the last tripleo meeting really on may 05th?09:09
jistri think it's fairly possible09:10
jistrit happens bi-weekly nowadays afaik09:10
jistrand the one after may 05 collided with summit09:10
mariosjistr: yeah kinda suspected/inferred that from last few (thanks for confirming), the biweekly thing09:11
mariosjistr: cool ta09:11
*** akrivoka has quit IRC09:12
*** akrivoka has joined #tripleo09:13
*** paramite has quit IRC09:13
*** paramite has joined #tripleo09:14
shardy is ready to land I think, CI is green and all the other pieces have now landed09:15
shardyIt enables stepped deployment via devtest using heat hooks09:15
shardyany reviews would be most appreciated :)09:15
gfidenteshardy, I'd merge it actually09:20
gfidenteI think I reviewed but forgot to vote the other day09:20
shardygfidente: great, thanks!09:21
gfidenteI had to do it to understand why you downvoted the step name/number matching :P09:22
openstackgerritMerged openstack/tripleo-incubator: Adding stepped deployment to
*** paramite has quit IRC09:22
jistryeah lgtm as well. I thought 30 secs might be a bit long time to wait between checks by default but with "-n 5" it's understandable09:22
gfidentejistr, I am running locally with 4G to see what happens09:23
shardygfidente: Yeah, we may want to introduce a little test for the templates to help enforce that naming convention, or it'll inevitably keep getting broken09:27
shardyassuming folks actually find the stepped deployment useful that is, it was flagged as a "must have" feature, but I get the impression nobody is really using it :\09:27
*** bvandenh has joined #tripleo09:28
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci: Add pacemaker status to host_info.txt
jistrgfidente: ^ that could help us see what is going on in there09:28
gfidentejistr, yeah I remember telling derekh I wanted to do it ... :)09:29
gfidentederekh, ^^ pcs in host_info that is09:29
gfidenteI even wanted to add ceph status :)09:29
*** regebro has joined #tripleo09:31
*** paramite has joined #tripleo09:34
*** bvandenh has quit IRC09:36
gfidentejistr, your turn for OPM, to rebase tht I think we need to update at least memcached and mongodb modules09:38
*** pdb has quit IRC09:40
openstackgerritMerged openstack/os-collect-config: Drop use of 'oslo' namespace package
*** pdb has joined #tripleo09:40
gfidentemarios, in,cm as local_ip I think you want hiera('conroller_host') not the list of controllers09:41
*** jprovazn has quit IRC09:56
gfidenteanyone got seed with disk full?10:02
gfidentemysql taking 20G for me there10:11
gfidenteof -bin files10:12
openstackgerritMerged openstack/diskimage-builder: rhel-common element should not use attach with activation key
mariosgfidente: thanks, been trying to parse like $neutron_ovs_ml2_agent_local_ips = $controller_node_ips.filter |$x| { $x != '' } :) will try that next10:14
*** paramite is now known as paramite|afk10:16
gfidentethat is despite all database summing ~100mb together10:17
*** u_glide has joined #tripleo10:20
u_glideHello folks! Thanks for great project! Is it possible to isolate somehow code for different distros in  diskimage-builder element?10:22
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Adds neutron-server and agents as pacemaker resources
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Switch CinderEnableIscsiBackend to CinderEnableLvmBackend
*** dtantsur|brb is now known as dtantsur10:38
jistrre OPM, ack10:43
gfidenteso for mysql/seed issue I had to do a 'mysql reset master' to delete old binglogs10:47
gfidentenot sure if we should set some expiry there10:47
jistrgfidente: mongodb should be ok i think
jistrgfidente: that's 2nd latest commit10:47
*** yog__ has quit IRC10:48
jistrthe latest commit in mongodb is related to puppet 4 unit tests so i doubt we need that10:48
gfidentejistr, ack, so I can only remember of memcached10:48
jistrand i'll also check if the keystone / glance / cinder updates landed10:49
jistr(db sync ones)10:49
jistri saw lbezdick submitted them10:50
jistrbut some of his patches are in merge conflict now10:50
hewbroccaderekh: you have an answer for u_glide ^^^10:54
derekhhewbrocca: sorry missed it, reading up10:54
*** jprovazn has joined #tripleo10:55
*** ishant has quit IRC10:59
derekhu_glide: when you say isolate code for different distros what is it your trying to achieve? Most of the distro specific elements are in their own element e.g.
derekhgfidente: RE: pcs commands in host info, iirc I had no problem with doing it did I?11:00
gfidentederekh, you didn't I just never pushed the change :)11:00
gfidentehoped you to vote, not to comment :P11:00
gfidentemerge merge merge11:01
derekhgfidente: cool, just making sure you weren't blocked on me or something +111:01
derekhahh is their a review up there11:01
gfidenteoh no no yeah I wanted to merge it11:01
gfidenteand I remember we discussed so was asking for +11:01
derekhgfidente: found it, looking11:02
*** bvandenh has joined #tripleo11:03
openstackgerritMerged openstack-infra/tripleo-ci: Add pacemaker status to host_info.txt
*** AzherKhan has quit IRC11:09
*** kodiakfff has quit IRC11:12
*** kodiakfff has joined #tripleo11:14
gfidentejistr, think I reproduced the nova issue locally11:14
gfidentepacemaker blocking nova chain on novncproxy not starting11:15
gfidentenova-novncproxy[2385]: AttributeError: 'module' object has no attribute 'ProxyRequestHandler'11:15
jistrthat's actually great news! i mean certainly it's better than just hitting that on CI11:15
gfidentewhy should that happen only in the HA job though :!>!_ >!KOPK!) ?11:16
*** bvandenh has quit IRC11:16
*** masco has quit IRC11:18
* jistr looking further11:18
* gfidente too11:19
jistrgot it11:19
jistrit's this
jistr[root@ov-i7rc3g7pal4-0-zb74lymu7bdd-controller-bt22c2vran5g ~]# rpm -q python-websockify11:20
gfidenteyeah same which made this fail11:20
gfidenteto me11:20
gfidenteI am still unsure why HA only?11:20
jistryeah i dunno... i just looked into non-HA and it has the same old websockify11:21
*** panda has quit IRC11:21
*** panda has joined #tripleo11:21
gfidentewhich asking yum it seems we're getting from the official fedora reppo11:22
openstackgerritJan Provaznik proposed openstack/tripleo-heat-templates: Config & deployments to update overcloud packages
hewbroccawoooo jprovazn and the overcloud updates!11:22
jprovaznhewbrocca: this is steve's patch ;)11:23
jprovaznhewbrocca: I'm just trying to make CI happy11:23
jprovaznwhich is pretty tough job11:23
hewbroccaLOL OK11:23
gfidentejistr, it's not only HA job11:24
jprovaznmaybe some other tripleoers more familiar with recent changes might help me to find out why is failing?11:24
gfidentejistr, HA job is the only one noticing the failure thanks to pacemaker I think11:24
jprovaznit's apparently because mysql on controller node failed to start because it didn't get heat metadata11:25
jprovaznbut I wonder why?11:25
gfidentefor the first time ever we have HA job doing something useful!11:25
gfidentewith thanks to pacemaker!11:26
gfidenteI like this11:26
* hewbrocca applauds11:27
gfidentejprovazn, you mean this
gfidentejprovazn, sorry, this  No hosts in Heat, nothing written.11:27
jprovazngfidente: no hosts in heat?11:28
jprovaznnot sure what you mean?11:28
gfidentethat seems to be it
*** kodiakfff has quit IRC11:29
jprovazngfidente: ah, yes - I focused on the 255 code11:30
jprovazngfidente: so it's a known bug?11:30
gfidenteI dunno11:30
* gfidente checking other jobs11:30
jistrjprovazn: see
jprovazngfidente: I already did - other jobs recently CId are green11:31
jistrjprovazn: the controller/compute resources never got created11:31
jistrjprovazn: puppet didn't even start running anywhere11:31
*** thrash|g0ne is now known as thrash11:31
jistrjprovazn: could be that there weren't enough resources available11:31
gfidentejprovazn, this is recent and doesn't log that
*** kodiakfff has joined #tripleo11:33
jprovaznjistr: wait, controller/compute resources - you mean nova instance resources or Controller/Compute ResourceGroup resources?11:35
jistrjprovazn: oh actually, it might not be not enough resources... i meant overloaded CI11:36
jistrjprovazn: sometimes it takes a looong time for the VMs to get created (can be 1 hour no problem), that's why we had to bump up heat timeout on HA job to 90 minutes11:36
jistrpuppet runs for ~ 13 minutes but machine reservation takes 1 hour11:37
jprovaznjistr: well, I can see that VMs spawned properly, I see logs from both - here is the log from controller where is the reason why it timedout -
jistri think i was lead astray by a false assumption that the compute/controller resources weren't changed in the patch, which is not true11:37
jprovaznjistr: but I don't know where to find in logs the reason it didn't find for what it was looking for :(11:38
* gfidente recheck everything failing on non-puppet :)11:38
jprovaznand because it failed twice with same error and other recent CI jobs pass, I almost tend to think it might be caused by the patch11:39
jprovaznwhich would really break my Friday mood11:39
jistryeah that's probable actually11:39
* hewbrocca sends jprovazn a beer to restore his friday mood11:43
jprovaznhehe, thanks11:43
*** rhallisey has joined #tripleo11:43
*** paramite|afk is now known as paramite11:43
*** lucasagomes is now known as lucas-hungry11:44
jprovaznactually that is a good idea - I don't think this is a problem which couldn't be solved by increased amount of beer11:44
jistr[2015-05-28 18:01:49,402] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]11:44
jistrthis looks like a problem right?11:44
jprovaznjistr: I suppose this is a first run when not metadata were avail yet, if you search for "ERROR" you can see that later it got further to the mysql setup script11:47
jistrah ok11:48
jistr2 questions/observations11:52
jistr1) should we run first puppet and then yum update?11:52
gfidentejistr, failing job is non-puppet11:53
jistrok, but the question was a bit more general in the sense -- does that patch implement what was intended?11:54
hewbroccaThere are not many problems which fail to yield to beer11:54
jistri saw a mailing list discussion where i think the outcome was that puppet should be run first, so that it knows which service packages got updated and which services need restarting11:55
jistr(not sure if i read all the latest messages but at some point at least this seemed like a consensus)11:55
*** trown|outttypeww is now known as trown11:56
jistrif that's the target, then i think the yum update needs to be part of controller-post(-puppet) instead of controller(-puppet)11:56
jistr2) does the overcloud in CI have internet connectivity? if not, could it be the cause of the issues with the yum update part?11:57
hewbroccayum update, in CI11:58
hewbroccaWHAT HAVE WE DONE11:58
* hewbrocca sends more beer11:58
hewbroccahey, does anybody know what we do as far as deploying keystone with httpd, or not?12:00
*** dprince has joined #tripleo12:01
hewbroccaIOW are we using Keystone v3 API with httpd and WSGI12:01
jistri don't think we deploy it with httpd atm12:01
hewbroccaor earlier keystone API with eventlet?12:01
openstackgerritMarek Aufart proposed openstack/python-tuskarclient: Filter and format parameters for plan role in OSC
jistrhewbrocca: i think it's v2 with eventlet12:02
hewbroccafabbione: ^^^12:03
hewbroccayou had asked about keystone...12:03
gfidentedirectxman12, what do we do with websockify for nova?12:05
gfidentedirectxman12, it looks like f22 will get 0.6 but f21 will stay with 0.5.1 ?12:05
gfidentedirectxman12, I can see 0.6 on rhel 7.1 as well though12:07
jistrgfidente: yeah i wanted to get back to this (too many issues for one friday). I have a deployment predating the Nova-pcmk patch, and nova-novncproxy is down there too, except nothing checks it so nothing cares, you were right that nothing new is broken, it just doesn't deploy because pacemaker cares about the breakage :)12:08
gfidentejistr, yeah I was thinking to comment with a FIXME in the .pp as wel12:08
gfidentejistr, or maybe a TODO12:08
gfidentejistr, I have yet to decide12:09
jistrgfidente: yeah i think the best we can do now is just disable novncproxy for right now and proceed12:09
gfidenteack will update12:09
*** jayg|g0n3 is now known as jayg12:09
gfidenteI am concerned though about f21 not getting 0.6 though12:09
gfidentedirectxman12, is owner, we'll see when he is back
fabbionehewbrocca: ack12:14
hewbroccafabbione: I responded to your email as well12:16
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Nova as Pacemaker resource
*** weshay has quit IRC12:24
marios\o/ green ha job12:30
marioshewbrocca: this means i am relinquished of my obligation to perform ritual harakiri by sundown local12:30
marioswin win12:30
hewbroccahot damn son12:31
hewbroccanice job12:31
gfidentemarios++ :)12:31
* hewbrocca thwacks marios on the... back12:31
mariosactually it was mostly gfidente so, for now,  i'll call the hit off on his head12:31
gfidente<hewbrocca> everybody shares the credit12:32
*** morazi has joined #tripleo12:32
*** weshay has joined #tripleo12:34
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add metering agent as neutron-server plugin
u_glidederekh: I want to achieve something similar to separation for different install types like here but for distros12:41
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Add neutron-lbaas as neutron-server plugin & neutron-lbaas-agent
hewbroccaholy christ marios lbaas? you really are going for broke aren't you12:42
jistrmarios: pls let me know when you open the bug discussed on the neutron review (the line there overrides the networking code, it's a bit weird that it didn't work for you locally, we should try to remove it in a follow up patch)12:43
jistrmarios: cause i'm ready to +a12:43
mariosjistr: sure, sec12:44
mariosjistr: where does it belong though? puppet-neutron? tht?12:44
gfidentemarios, yeah I think it is obvious that for other classes we inherit the params12:44
gfidenteso it's something with that which seems broken12:45
jistrmandre: tht12:45
jistrmarios: tht12:45
jistrmandre: sorry again :)12:45
gfidentejistr, tht?12:45
marios'what does mine say'12:45
u_glideok, another question - folks, Is it possible to specify custom parameter in package-installs.yaml/json ? like --auto-remove for apt-get ?12:45
jistrmarios, gfidente: are we talking about this thing, right?
jistrmarios, gfidente: that line needs to be removed from tht, so the bug belongs to tht12:46
mariosjistr: ok, will file there for now anyway12:46
gfidentejistr, marios sec12:46
gfidenteI think the line in tht is fine, module does not seem to be using it when defining clas { ... }12:47
gfidentebut that shouldn't be the case, so I think the bug goes to puppet-neutron12:47
jistrok sorry folks :)12:47
gfidentebut actually, we need to track this issue in the yaml/pp couple as well12:48
gfidenteso probably best to add both components to same bug12:48
jistryeah so both ideally12:48
*** lucas-hungry is now known as lucasagomes12:48
*** regebro has quit IRC12:50
openstackLaunchpad bug 1460058 in tripleo "tripleo-heat-templates hiera neutron::agents::ml2::ovs::local_ip defined and passed but not consumed during puppet pacemaker run" [Undecided,New]12:53
mariosgfidente: jistr yeah wherever as long as we have a placeholder (incidentally, tht bugs go against tripleo right? can't see tht specific bugs_12:54
*** radez_g0n3 is now known as radez12:54
mariosgfidente: feel free to change it to whatever12:54
openstackgerritMerged openstack/tripleo-heat-templates: Adds neutron-server and agents as pacemaker resources
*** lblanchard has joined #tripleo12:56
jistrmarios: thanks! merged12:57
marioshewbrocca: can we all retire now?12:59
hewbroccaWe are DONE13:00
mariosgfidente: how can i help with the nova one. is it still useful to have a run with that? (will you rebase :( sorry it might be a tricky one, i can do it for you? /me checks)13:01
gfidenteno worries, rebasing that and a couple of others on top so we don't conflict13:02
gfidentewe'll have to wait for CI results anyway13:02
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Nova as Pacemaker resource
mariosgfidente: will still hit you. just for fun, not because it is your fault.13:03
gfidenteI will never ever make any useful comment anymore13:04
* gfidente wasn't this already the case?13:04
mariosgfidente: well if you're gonna be like that about it i'm going to go pull master now that _my_ patch merged and then try out your patch. and there is nothing you can do about it.13:05
gfidentemarios, empathy was on 'useful' not on 'comment'13:06
gfidentewhich was already the case13:06
mariosy u hate me?13:06
gfidentenot if we merge the things13:07
derekhu_glide: for code seperation based on distro you can use $DISTRO_NAME e.g. the kind for thing you'll see in
u_glidederekh: awesome! thanks!13:08
mariosgfidente: i rebuilt images like an hour ago (puppet modules from source) should be good yes?13:08
dprinceshardy: just discovered that parameter_defaults doesn't apply to the top level stack. It only applies to nested stacks...
openstackLaunchpad bug 1460063 in heat "parameter_defaults only apply to nested stacks" [Undecided,New]13:09
gfidente :)13:09
dprinceshardy: can work around it for now. Any thoughts on what I'm suggesting?13:09
dprinceshardy: my choices are to duplicate things into both parameter_defaults and parameters... or I can be sneaky and return ServiceNetMap to the top level stack from one of the nested stacks.13:11
dprinceshardy: I'm leaning towards the latter so as not to force us to duplicate things in our env files for now...13:12
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Heat as a Pacemaker resource
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Nova as Pacemaker resource
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Ceilometer as a Pacemaker resource
gfidentemarios, look what I rebased for you13:18
shardydprince: We should just fix it in heat, I've taken the bug and will aim to get a patch to you by EOD13:22
shardywhatever works in the meantime so it doesn't hold you up :)13:22
*** noslzzp_ has joined #tripleo13:22
shardydprince: you could probably use a yaml reference between parameter_defaults and parameters as a workaround?13:23
dprinceshardy: could try that too13:23
*** noslzzp has quit IRC13:25
*** jeh has joined #tripleo13:29
*** subscope has quit IRC13:35
*** pblaho has quit IRC13:38
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Make all-nodes Ip networks configurable
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add virtual IPs for split out networks
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Combine Heat API networks into single net
*** gfidente is now known as gfidente|afk13:49
*** paramite has quit IRC13:50
*** subscope has joined #tripleo13:51
*** tzumainn has joined #tripleo13:57
*** mcornea has quit IRC14:00
dasmguys, quick question about HA for controllers. Does Tripleo have it?14:00
dasmi'm a little confused, because afair i've seen somewhere info, that tripleo allows to deploy cluster of controllers in ha. but now cannot find it anymore.14:01
dasmand i see ongoing work in ha of services, by pacemaker.14:01
dasmso, in short words: what is the status of ha ;)14:01
hewbroccadasm: you're asking at exactly the right time!14:02
*** mcornea has joined #tripleo14:02
hewbroccaI believe we have merged just about all of the pacemaker resource support14:02
dasmhewbrocca: i see heat, ceilometer and nova in review, so probably some bits are still to be merged.14:03
hewbroccawhich means that we, as of this afternoon, support three-controller HA with Pacemaker, Galera, Rabbit, Neutron, and so on14:03
hewbroccaheat, ceilo, and nova are the last bits waiting to be merged14:03
hewbroccaCeph, too!14:03
dasmso, may i assume that at Morning/Tuesday we'll have full HA support in Tripleo? Is it correct?14:04
dasm(so what am i doing here? :P)14:04
hewbroccadasm: everything but automated config of fence agents14:06
hewbroccawhich we will be working on next week14:06
dasmhewbrocca: thanks, you helped me very much.14:07
hewbroccaThe other big thing we're trying to wrap up is network isolation14:09
hewbroccawe're getting there, but there is still quite a bit to do14:09
hewbroccadprince: is leading that work14:09
*** cody-somerville has quit IRC14:13
*** subscope has quit IRC14:15
*** sseago has left #tripleo14:22
jprovaznshardy: slagle: CI is finally happy with , could you please take a look when you have a minute?14:23
*** mcornea has quit IRC14:24
*** mcornea has joined #tripleo14:25
* jistr reloc14:26
*** jistr has quit IRC14:26
*** olaph has quit IRC14:29
*** cody-somerville has joined #tripleo14:30
*** subscope has joined #tripleo14:30
*** olaph has joined #tripleo14:30
dasmhewbrocca: mhm.. good to know about dprince work.14:35
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Configure virtual IPs for split out networks
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Use node IP lists for HA Proxy ipaddresses
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Add interface_for_ip function
hewbroccaholy hell dprince, VIPs?14:38
hewbroccayou are a machine14:38
dprincehewbrocca: well, I haven't touched the pacemaker bits yet. but this is a start14:40
dprincehewbrocca: this is coming together14:40
*** dasm is now known as dasm|afk14:42
*** rameshg87 has joined #tripleo14:42
rameshg87hello tripleo-cores14:43
rameshg87can someone please have a look at and
rameshg87it has been with one +2 for more than 20 days14:44
*** rameshg87 is now known as rameshg87-away14:45
*** sdake has joined #tripleo14:46
*** jistr has joined #tripleo14:47
hewbroccadprince: Yes... yes it is...14:49
*** rameshg87-away has quit IRC14:52
*** gfidente|afk is now known as gfidente15:03
*** jprovazn has quit IRC15:07
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Heat as a Pacemaker resource
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Nova as Pacemaker resource
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Add Ceilometer as a Pacemaker resource
mariosgfidente: why you -1 your nova15:15
mariosah looking again15:15
gfidenteto remove nova-cert15:15
gfidentethere was some conversation about it not being in the refarch15:16
gfidenteso I removed it and removed the dependency on metering and lbaas15:16
*** rameshg87 has joined #tripleo15:17
*** rameshg87 has left #tripleo15:18
*** panda has quit IRC15:21
*** alop has joined #tripleo15:21
*** panda has joined #tripleo15:21
gfidentejistr, L3HA in CI
*** lazy_prince has quit IRC15:24
*** sdake_ has joined #tripleo15:26
gfidentemarios, ^^15:26
*** alop has quit IRC15:27
*** ifarkas has quit IRC15:28
*** sdake has quit IRC15:30
*** bvandenh has joined #tripleo15:30
*** subscope has quit IRC15:35
jistrgfidente: merged15:41
jistrgfidente: later i realized that we could even put that yaml to t-h-t but it's not worth a resubmit and re-CI15:42
openstackgerritMerged openstack-infra/tripleo-ci: Enable NeutronL3HA for the HA job
gfidentejistr, you mean using a specific environment file for CI?15:43
mariosHAPPY FRIDAY15:44
jistrmarios: o/ enjoy the weekend15:44
*** mcornea has quit IRC15:45
jistrgfidente: not really, i meant just instead of cat <<EOF we can just put the l3ha file in t-h-t proper, and then folks can use it even with their dev envs by just pointing to it (in combination with the HA file, exactly as TOCI has it)15:45
jistrjust too many "just" in one sentence15:46
gfidentejistr, I see, so not a specific file for CI but one with L3HA settings only to be combined with HA?15:46
jistrgfidente: yup15:46
gfidenteI promise if you don't do, I might do it sometime next week :)15:47
jistryeah it's a nice thing to have but low prio15:47
*** alop has joined #tripleo15:48
mariosgfidente: i was forced to reveal the truth15:49
mariosabout how you treat me15:49
*** daneyon has joined #tripleo15:50
gfidenteI think this has to be sorted somewhere not on irc15:50
*** cwolferh has joined #tripleo15:51
mariosgfidente: i am going to leave now. but i'll leave my nick around to keep an eye on you. and i won't respond. i will ignore you. until monday. and there is nothing you can do about it15:51
*** daneyon_ has joined #tripleo15:53
*** daneyon has quit IRC15:55
shardyIs there any info about the ceph implementation around other than the templates?15:56
jistrmarios, gfidente: LOL you two15:56
jistri'm off, have a nice weekend everybody15:57
shardyI'm looking for details of how it all hangs together with the Block/ObjectStorage nodes15:57
hewbroccammm morazi ^^^15:57
hewbroccawhere do we find that15:57
hewbroccashardy: are you blogging, or writing patches?15:58
*** jistr has quit IRC15:58
shardyhewbrocca: slide preparation, looking for some cut/pasteable resources ;)15:58
hewbroccaahhh -- noslzzp_ ^^^ ?15:59
noslzzp_(FBI call starting)15:59
gfidenteshardy, objectstorage is not using ceph yet15:59
shardyIt's entirely possible no such resources exist, I just wanted to avoid wheel reinvention :)15:59
shardygfidente: Ok, thanks what I wanted to know16:00
gfidenteto be honest the documentation we had was in the cinder/ha blueprint16:00
shardybut the cinder backend gets automagically switched to ceph right?16:00
gfidenteautomagically no16:00
gfidenteI think best documentation we have is this env file16:00
shardygfidente: Aha, thanks I'd forgotten that16:02
gfidenteso you can optionally enable it for glance/cinder/nova16:02
gfidenteand optionally deploy osd on controller16:02
gfidenteinstack is enabling ceph for all three when cephstoragescale > 116:03
shardygfidente: perfect, thanks - that's exactly the info I needed16:03
gfidenteyeah so instack is automagically ...16:04
gfidenteshardy, spare a vote on CI! :)16:05
*** jistr has joined #tripleo16:06
shardygfidente: done16:08
gfidentetks, it's there from a while now16:08
gfidentewould be nice to have it16:08
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add virtual IPs for split out networks
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Add PortName to ports stacks
*** jistr has quit IRC16:09
*** tteggel has quit IRC16:09
shardygfidente: one last dumb ceph question, with CinderEnableRbdBackend do you still deploy BlockStorage nodes, or do the cinder services on the controller interact directly with ceph?16:10
* shardy needs more hardware to try this stuff..16:10
gfidenteshardy, BlockStorage nodes are meant to run cinder-volume only and they only get configured with the LVM backend16:10
gfidenteshardy, CephStorage nodes run ceph-osd16:11
*** tteggel has joined #tripleo16:11
gfidenteso the two scale independently and you can have 0 or more of each16:11
gfidentethe rbd driver talks to both ceph-mon (running on controller) and ceph-osd (running on cephstorage nodes or on controller)16:12
*** dtantsur is now known as dtantsur|afk16:12
shardygfidente: Ok, thanks - that clarifies things, I wasn't sure exactly how the services were wired up without digging in templates16:13
morazishardy, I think gfidente has got you mostly covered there and agree re: object storage.  that would come in when we do the rgw work, but it isn't there yet.16:13
*** rwsu has joined #tripleo16:15
*** Goneri has quit IRC16:20
*** spzala has quit IRC16:24
*** spzala has joined #tripleo16:25
*** rhallisey has quit IRC16:34
*** rhallisey has joined #tripleo16:39
dprincedsneddon: How would you feel about naming KeystonePublicApiNetwork just KeystoneApiNetwork.16:39
dprincedsneddon: it will have VIPs (configurable of course) bound to 2 networks, public, and say internal_api by default.16:40
dprincedsneddon: the public network is sort of a given however, we only need to specify the "internal_api" network via this parameter...16:41
openstackgerritOpenStack Proposal Bot proposed openstack/os-collect-config: Updated from global requirements
dprincedsneddon: in other words, I'm just saying I find the 'Public' in that setting confusing... because we probably want it set to internal_api by default.16:41
* gfidente salute everyone16:42
*** gfidente has quit IRC16:43
*** derekh has quit IRC16:48
*** weshay has quit IRC16:53
*** weshay has joined #tripleo16:53
openstackgerritMerged openstack/diskimage-builder: Fix disk image create errors behind proxy
*** spzala has quit IRC16:55
*** spzala has joined #tripleo16:55
*** jeh has quit IRC16:56
*** daneyon_ has quit IRC17:05
dsneddondprince, Keystone Public Network should be on the External network, not internal_api.17:08
*** sdake has joined #tripleo17:09
dprincedsneddon: we use the keystone API both internally and externally.
dprincedsneddon: this setting will effectively control the private side (i.e. swap in for $controller_virtual_ip in that puppet manifest)17:10
dprincedsneddon: Most of the OpenStack API's already have VIPs on the "public" network...17:11
*** sdake_ has quit IRC17:12
*** noslzzp has joined #tripleo17:12
*** noslzzp_ has quit IRC17:13
dsneddondprince, We aren't making use of internal URLs at this time, only Public. The OpenStack Security Guide is confusing on this matter and recommends Internal URLs to be "forward-compatible":
*** sdake_ has joined #tripleo17:20
dprincedsneddon: right, so that is what we do via the catalog. But within the cluster we are using some "private" IPs directly17:20
dsneddondprince, We definitely want HAProxy listening on both the Public network and the Internal API network.17:21
dprincedsneddon: it is for these cases that I find the KeystonePublicApiNetwork confusing17:21
dprincedsneddon: I think we are on the same page... all I'm dealing with here is a naming/confusion issue17:21
dprincedsneddon: I want the templates to read clearly....17:21
dsneddondprince, I don't see how we can combine KeystoneApiNetwork and KeystonePublicApiNetwork and still be able to control the placement onto two places. Remember, in the end we want Keystone listening on port 5000 in two places.17:23
*** sdake has quit IRC17:23
dprincedsneddon: it will always listen on the public one. The other is what I'm suggesting is currently configurable.17:23
dsneddondprince, OK, that's fine.17:24
dprincedsneddon: so given that would you buy calling it just KeystoneApiNetwork then (drop the Public)17:24
dprincedsneddon: and we'd still have the separate setting for KeystoneAdminApiNetwork too17:25
dprincedsneddon: perhaps KeystoneAdminApiNetwork is the one we should potentially not have listening on public BTW...17:25
*** cody-somerville has quit IRC17:25
dprincedsneddon: or at least an option to do that if someone wants it17:25
*** sdake has joined #tripleo17:26
dsneddondprince, I could see making the KeystoneAdminApiNetwork configurable, but most people are not going to want the Keystone Admin API to be public-facing, because that's where you add users/passwords.17:26
*** barra204 has quit IRC17:26
dsneddondprince, Also, the Admin API doesn't seem to be wired up, or at least I didn't find where it was. I've been meaning to talk to you about that.17:27
dprincedsneddon: well, it currently would do both public (always) and the other is configurable:
*** sdake_ has quit IRC17:29
dsneddondprince, We should change this, but we can do a follow-up patch. The Admin API shouldn't be enabled on the Public by default.17:30
dprincedsneddon: okay, sounds reasonable17:30
*** trown is now known as trown|lunch17:31
*** penick has joined #tripleo17:41
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Configure virtual IPs for split out networks
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Add interface_for_ip function
dsneddondprince, Hoo boy, MySQL should definitely not be on the Public API network. How about a follow-up security-related patch that cleans up several of these things?17:44
dhellmannbnemec: thanks17:44
dprincedsneddon: mysql is only on the "private" network currently. So I think we are good there:
dhellmannslagle: I would like to release os-collect-config 0.1.35 on Monday to include the change that unblocks the work we're doing to drop the use of the oslo namespace package. I can tag the release, but wanted to coordinate with you to ensure that it wouldn't cause you any trouble.17:45
dprincedsneddon: so once these patches land essentially it will only bind to its VIP on the configured MysqlNetwork17:45
dprincedsneddon: a security audit would be in order when we are done this stuff though...17:46
dsneddondprince, If that's the case, then what is happening in L545 of this file:
dsneddondprince, Oops I mean L56317:47
dprincedsneddon: controller_virtual_ip is the provisioning network17:47
dsneddondprince, Actually, I think you made a typo. You turned [controller_virtual_ip] into unique([hiera('mysql_vip', $controller_virtual_ip), $public_virtual_ip])17:48
*** noslzzp_ has joined #tripleo17:49
dprincedsneddon: Oh, Yeah. That is nasty. -1 me for that17:49
dprincedsneddon: copy paste error17:49
bnemecdhellmann: slagle is out today.  I don't think that will be a problem though since that's the only change in the project since the last release.17:51
*** noslzzp has quit IRC17:51
dhellmannbnemec: so it's ok for me to release?17:52
bnemecdhellmann: Yeah, should be fine.  I don't think there's anything significantly different in our release process.
bnemecWe should probably get rid of that and just use the new release tools.17:55
dhellmannbnemec: yeah, I was planning to use the library release scripts, but I'll check and make sure there are no big differences in process17:55
dhellmannoh, no, cool, that's just the manual process that has been automated now17:56
bnemecYeah, it's not doing anything too crazy.17:56
*** cwolferh has quit IRC17:56
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Configure virtual IPs for split out networks
*** aufi has quit IRC18:00
*** spzala has quit IRC18:03
*** spzala has joined #tripleo18:03
*** penick has quit IRC18:05
*** jrist has quit IRC18:06
*** sdake_ has joined #tripleo18:07
*** barra204 has joined #tripleo18:08
*** sdake has quit IRC18:12
*** jrist has joined #tripleo18:12
*** barra204 has quit IRC18:15
*** pradk has joined #tripleo18:15
*** penick has joined #tripleo18:17
*** spzala has quit IRC18:18
*** spzala has joined #tripleo18:18
*** barra204 has joined #tripleo18:19
*** spzala has left #tripleo18:19
*** barra204_ has joined #tripleo18:27
*** pradk has quit IRC18:27
*** pino|work has quit IRC18:30
*** pradk has joined #tripleo18:30
*** barra204 has quit IRC18:31
*** trown|lunch is now known as trown18:36
*** dsneddon is now known as dsneddon_lunch18:37
*** barra204_ is now known as shakamunyi18:52
*** jcoufal has joined #tripleo18:54
*** jcoufal has quit IRC18:55
*** penick has quit IRC19:00
*** panda has quit IRC19:21
*** panda has joined #tripleo19:22
*** lucasagomes has quit IRC19:24
*** penick has joined #tripleo19:33
*** yamahata has joined #tripleo19:35
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Configure virtual IPs for split out networks
openstackgerritDan Prince proposed stackforge/puppet-tripleo: Add interface_for_ip function
*** yamahata has quit IRC19:43
*** sdake_ is now known as sdake19:43
*** yamahata has joined #tripleo19:43
*** dprince has quit IRC19:44
*** sdake_ has joined #tripleo19:46
*** sdake has quit IRC19:49
*** sdake has joined #tripleo19:49
*** penick has quit IRC19:50
*** sdake_ has quit IRC19:53
*** cwolferh has joined #tripleo19:55
openstackgerritMerged openstack/tripleo-heat-templates: Add Nova as Pacemaker resource
*** openstack has joined #tripleo20:06
*** openstack has joined #tripleo20:31
*** sdake_ has joined #tripleo20:36
*** lblanchard has quit IRC20:39
*** sdake has quit IRC20:40
*** spzala has joined #tripleo20:49
*** sdake has joined #tripleo20:50
*** dsneddon_lunch is now known as dsneddon20:53
*** sdake_ has quit IRC20:54
*** openstackgerrit has quit IRC20:59
*** openstackgerrit has joined #tripleo20:59
*** untriaged-bot has joined #tripleo21:00
untriaged-botUntriaged bugs so far:21:00
openstackLaunchpad bug 1459465 in tripleo "devtest fails to build ramdisk on Fedora 21 due to Yum multilib error" [Undecided,New]21:00
*** untriaged-bot has quit IRC21:00
*** pradk has quit IRC21:02
*** julim has quit IRC21:07
*** trown is now known as trown|outttypeww21:09
openstackgerritMerged openstack/tripleo-heat-templates: Use heat inputs for network port settings
openstackgerritimain proposed openstack/tripleo-heat-templates: Containerized TripleO Overcloud.
openstackgerritMerged openstack/tripleo-heat-templates: Map Mysql to isolated networks
*** rhallisey has quit IRC21:23
*** sdake is now known as steakache21:23
*** steakache is now known as sdake21:24
*** shakamunyi has quit IRC21:26
*** openstack has joined #tripleo21:32 [freenode-info] channel trolls and no channel staff around to help? please check with freenode support:
*** openstackstatus has joined #tripleo21:32
*** ChanServ sets mode: +v openstackstatus21:32
*** Goneri has joined #tripleo21:33
*** stendulker has quit IRC22:03
*** Goneri has quit IRC22:03
*** jayg is now known as jayg|g0n322:06
openstackgerritOpenStack Proposal Bot proposed openstack/tuskar: Updated from global requirements
*** cody-somerville has joined #tripleo22:32
*** cody-somerville has quit IRC22:36
*** sdake_ has joined #tripleo22:39
*** sdake has quit IRC22:43
*** sdake has joined #tripleo22:49
*** sdake has quit IRC22:49
*** sdake has joined #tripleo22:49
*** sdake_ has quit IRC22:52
*** radek__ has quit IRC22:54
*** penick has joined #tripleo22:57
*** penick has quit IRC22:58
*** Goneri has joined #tripleo23:09
*** Marga_ has joined #tripleo23:10
*** Marga_ has quit IRC23:16
openstackgerritDan Sneddon proposed openstack/tripleo-incubator: Document how to use network isolation/static IPs
*** penick has joined #tripleo23:24
*** Goneri has quit IRC23:29
*** penick has quit IRC23:31
*** spzala has quit IRC23:50
*** AzherKhan has joined #tripleo23:54
*** alop has quit IRC23:55
*** sdake_ has joined #tripleo23:56

Generated by 2.14.0 by Marius Gedminas - find it at!