Monday, 2015-07-20

SpamapSgreghaynes: regarding the leftover .raw in 194574 .. that should not matter. unmounting tmpfs frees the space, and rm -r'ing it frees the space on non-tmpfs usage. So.. ?00:16
*** chlong has joined #tripleo00:19
*** shadower has quit IRC00:23
*** shadower has joined #tripleo00:23
*** goneri_ has quit IRC00:31
*** goneri_ has joined #tripleo01:05
*** lan has joined #tripleo01:39
lifelessSpamapS: what if you're building on an rPi ?02:08
*** goneri_ has quit IRC02:36
*** cwolferh has quit IRC02:50
*** stendulker has joined #tripleo03:21
*** noslzzp has quit IRC03:38
*** noslzzp has joined #tripleo03:39
*** noslzzp has quit IRC03:40
*** pbourke has quit IRC03:56
*** pbourke has joined #tripleo03:57
*** weshay has quit IRC04:02
*** tchaypo2 has quit IRC04:58
*** rhefner has quit IRC04:59
*** rhefner has joined #tripleo05:00
*** ramishra has quit IRC05:02
*** rhefner has quit IRC05:04
*** rhefner has joined #tripleo05:07
*** ramishra has joined #tripleo05:18
*** stendulker has quit IRC05:23
*** stendulker has joined #tripleo05:23
*** regebro has joined #tripleo05:26
*** chlong has quit IRC05:27
*** shivrao has joined #tripleo05:28
*** shivrao_ has joined #tripleo05:32
*** shivrao has quit IRC05:33
*** shivrao_ is now known as shivrao05:33
*** ramishra has quit IRC05:35
*** CaptTofu_ has quit IRC05:35
*** Ng has quit IRC05:35
*** h00327910_ has quit IRC05:36
*** BadCub has quit IRC05:36
*** tchaypo has quit IRC05:36
*** rhefner has quit IRC05:36
*** chlong has joined #tripleo05:40
*** ramishra has joined #tripleo05:41
*** CaptTofu_ has joined #tripleo05:41
*** hewbrocca has quit IRC05:41
*** BadCub has joined #tripleo05:43
*** rhefner has joined #tripleo05:44
*** tchaypo has joined #tripleo05:45
*** Ng has joined #tripleo05:47
*** ChanServ sets mode: +v Ng05:47
*** h00327910_ has joined #tripleo05:48
*** masco_ has joined #tripleo05:58
*** paramite has quit IRC06:04
*** lsmola has joined #tripleo06:12
*** hewbrocca has joined #tripleo06:13
*** rpodolyaka has quit IRC06:14
*** rpodolyaka has joined #tripleo06:15
*** chlong has quit IRC06:20
*** jtomasek has joined #tripleo06:20
*** shivrao has quit IRC06:25
*** ishant has joined #tripleo06:25
*** ishant|2 has joined #tripleo06:35
*** chlong has joined #tripleo06:36
*** ukalifon has joined #tripleo06:36
*** ishant has quit IRC06:38
*** akrivoka has joined #tripleo06:47
*** jprovazn has joined #tripleo06:48
*** bvandenh has joined #tripleo06:51
*** jtomasek has quit IRC06:54
*** jtomasek has joined #tripleo06:54
*** mcornea has joined #tripleo06:57
*** ishant has joined #tripleo06:58
*** ishant|2 has quit IRC07:02
greghaynesSpamapS: oh! right, we kill the whole dir.07:02
greghaynesyea, having a fallback to non tmpfs seems like a thing to do but I think its not needed for that to merge07:03
*** paramite has joined #tripleo07:05
*** athomas has joined #tripleo07:07
*** pino|work has joined #tripleo07:08
*** jcoufal has joined #tripleo07:11
*** ifarkas has joined #tripleo07:14
*** shardy has joined #tripleo07:15
openstackgerritMerged openstack/tripleo-heat-templates: Allow overlapping IPs in Neutron  https://review.openstack.org/20302007:20
*** yog__ has joined #tripleo07:28
*** BadCub has quit IRC07:31
*** CaptTofu_ has quit IRC07:31
*** tchaypo has quit IRC07:31
*** chlong has quit IRC07:34
*** paramite is now known as paramite|afk07:38
*** paramite|afk is now known as paramite07:40
*** dtantsur|afk is now known as dtantsur07:40
*** pcaruana has quit IRC07:42
*** yamahata has joined #tripleo07:48
*** jcoufal_ has joined #tripleo07:49
*** jcoufal has quit IRC07:51
*** BadCub has joined #tripleo07:53
*** jistr has joined #tripleo07:54
*** aufi has joined #tripleo07:58
*** bvandenh has quit IRC08:01
*** lucasagomes has joined #tripleo08:03
*** CaptTofu_ has joined #tripleo08:06
*** tchaypo has joined #tripleo08:07
*** jcoufal_ has quit IRC08:09
*** jcoufal has joined #tripleo08:15
*** derekh has joined #tripleo08:15
*** paramite is now known as paramite|afk08:16
*** bvandenh has joined #tripleo08:16
*** tchaypo has quit IRC08:21
dasmgood morning08:22
*** tchaypo has joined #tripleo08:25
*** athomas has quit IRC08:25
*** gfidente has joined #tripleo08:30
*** gfidente has quit IRC08:35
*** yamahata has quit IRC08:37
*** gfidente has joined #tripleo08:40
*** gfidente has quit IRC08:40
*** gfidente has joined #tripleo08:40
*** athomas has joined #tripleo08:41
*** pcaruana has joined #tripleo08:43
*** shardy_ has joined #tripleo08:44
*** shardy has quit IRC08:45
*** shardy_ has quit IRC08:49
*** shardy has joined #tripleo08:50
*** paramite|afk is now known as paramite08:57
gfidentejistr, this https://review.openstack.org/#/c/202804/ I tested it manually08:58
jistrgfidente: umm -- you mean it's related to some bz? or we should merge?09:02
jistror both :)09:02
*** sbelous has quit IRC09:03
gfidenteboth yes09:05
gfidenteI was surprised that those settings default to publicUrl09:06
gfidente:(09:06
gfidentebut the failure in that CI job is interesting09:07
*** pelix has joined #tripleo09:07
gfidenteapparently os-collect-config never got metadata from heat09:07
*** bvandenh has quit IRC09:08
*** paramite is now known as paramite|afk09:08
*** paramite|afk is now known as paramite09:10
jistrgfidente: which bz?09:11
gfidentehttps://bugzilla.redhat.com/show_bug.cgi?id=124401309:12
openstackbugzilla.redhat.com bug 1244013 in openstack-tripleo-heat-templates "compute nodes configured to talk to cinder api over publicurl" [Unspecified,On_qa] - Assigned to gfidente09:12
uvirtbotgfidente: Error: Could not parse XML returned by bugzilla.redhat.com: HTTP Error 404: Not Found09:12
*** dtantsur is now known as dtantsur|brb09:14
*** sbelous has joined #tripleo09:14
*** pblaho has joined #tripleo09:16
*** bvandenh has joined #tripleo09:22
jistrgfidente: interesting that it failed on a different job second time. I think the failures might be entirely unrelated to the patch...09:29
jistri'll give it one more recheck just to be sure09:29
gfidentejistr, was it same failure?09:30
jistrgfidente: nope, they fail on different nodes09:31
jistrone compute09:31
jistrthe other controller09:31
*** akrivoka has quit IRC09:32
gfidenteoh your guess about consoleauth was totally correcy09:32
gfidenteI missed entirely the memcache backends09:32
hewbroccagfidente: uh oh09:32
jistrgfidente: sorry, do you mean we have another consoleauth problem, or are you referring to the fix you've done before? (letting nova know about memchache)09:40
hewbroccaoh, maybe gfidente is referring to that last minute fix on Friday09:40
jistrah ok, then it's fine :)09:41
jistryeah it was a funny bug09:41
gfidentejistr, yeah but you figured they were not sharing the token09:41
gfidentejistr, so the change was right there09:42
gfidentejistr, and we didn't notice it was missing09:42
jistrhehe yeah09:42
*** regebro has quit IRC09:57
openstackgerritMerged openstack/diskimage-builder: Cleanup yum downloading  https://review.openstack.org/18662009:58
openstackgerritMerged openstack/diskimage-builder: Add base element for using docker as image base  https://review.openstack.org/18916509:59
*** regebro has joined #tripleo10:00
*** shardy_ has joined #tripleo10:03
*** pelix has quit IRC10:04
*** shardy has quit IRC10:04
*** pelix has joined #tripleo10:04
*** pelix has quit IRC10:05
*** pelix has joined #tripleo10:05
*** pelix has quit IRC10:06
*** shardy_ has quit IRC10:08
*** shardy has joined #tripleo10:09
*** pelix has joined #tripleo10:11
*** akrivoka has joined #tripleo10:28
*** dtantsur|brb is now known as dtantsur10:36
*** pelix has quit IRC10:37
*** pelix has joined #tripleo10:38
*** akrivoka has quit IRC10:38
*** mcornea has quit IRC10:47
*** paramite is now known as paramite|afk10:53
*** akrivoka|conf has joined #tripleo10:56
*** akrivoka|conf has quit IRC10:58
*** akrivoka has joined #tripleo10:58
*** paramite|afk is now known as paramite11:01
*** mcornea has joined #tripleo11:13
*** derekh has quit IRC11:14
*** lucasagomes is now known as lucas-hungry11:17
*** rhallisey has joined #tripleo11:23
openstackgerritMerged openstack/tripleo-heat-templates: Ensure compute nodes use internalURL as catalog_info  https://review.openstack.org/20280411:27
*** bvandenh has quit IRC11:30
*** bvandenh has joined #tripleo11:31
*** dprince has joined #tripleo11:35
*** regebro has quit IRC11:38
*** regebro has joined #tripleo11:39
*** stendulker has quit IRC11:39
*** julim has joined #tripleo11:56
*** jdob has joined #tripleo11:56
*** matbu has joined #tripleo11:58
*** rbrady has joined #tripleo11:58
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Drive DB initialization via Hiera  https://review.openstack.org/20360611:59
*** masco_ has quit IRC12:04
*** mcornea has quit IRC12:11
*** mcornea has joined #tripleo12:11
*** lucas-hungry is now known as lucasagomes12:11
*** jayg|g0n3 is now known as jayg12:12
*** thrash|g0ne is now known as thrash12:14
*** regebro has quit IRC12:21
*** regebro has joined #tripleo12:22
*** akrivoka has quit IRC12:24
*** weshay has joined #tripleo12:24
openstackgerritYanis Guenane proposed openstack/tripleo-image-elements: Default to latest rdo-release  https://review.openstack.org/20361112:27
*** matbu has quit IRC12:30
*** chlong has joined #tripleo12:34
openstackgerritAna Krivokapic proposed openstack/tuskar-ui: Helper script to set up a development environment  https://review.openstack.org/20022812:37
*** trown|outttypeww is now known as trown12:38
*** akrivoka has joined #tripleo12:43
*** regebro has quit IRC12:45
gfidentedprince, spredzy regarding https://review.openstack.org/#/c/203611/112:48
*** jprovazn has quit IRC12:48
gfidenteI think the reason why it doesn't have a default is that we wanted12:49
gfidenteto set dependencies on rdo-kilo, like rdo-icehouse and never use rdo-release12:49
gfidentelike https://github.com/openstack/tripleo-image-elements/tree/master/elements/rdo-icehouse12:49
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Drive DB initialization via Hiera  https://review.openstack.org/20360612:49
spredzygfidente, ok so we would create a new element for each openstack release ?12:50
gfidentespredzy, that was the idea12:50
spredzyrdo-liberty, rdo-kilo, rdo-icehouse, etc...12:50
gfidentebut I personally don't know if it's best thing12:50
spredzyso what is the point of rdo-release ?12:50
gfidenteit has the shared code12:50
gfidenterdo-{icehouse,...} only export the var actually12:51
dprincegfidente: right. I'm thinking perhaps the missing peice here is a validation to make sure it gets set somewhere. Either via an element or an ENV variable12:51
gfidenteI think I like more having a single element12:51
dprincegfidente: I would prefer not to have more elements for every release12:51
dprincespredzy: would a validation do instead?12:52
gfidenteyeah it's just abound finding a common a way and go for it12:52
dprincespredzy: I set RDO_RELEASE in my env12:52
gfidente*about12:52
spredzydprince, yup I know but the idea I wanted to implement bassically is that if you dont specify anything it will default to latest12:52
spredzyelse to the one specified12:52
spredzyincluding only one element the rdo-release one12:52
dprincespredzy: given we default Fedora to 21 or 22 in DIB.12:53
dprincespredzy: I think defaulting it to latest is probably reasonable12:53
spredzyyep, plus from a user perspective, saying I want rdo-release compononent on my images should do something12:53
spredzysomethting that can be changed by an envrionment variable, but it still should do something reasonable12:54
*** goneri_ has joined #tripleo12:54
spredzygfidente, ^12:54
spredzywdyt ?12:54
gfidenteto be honest, I think using only rdo-release with a sane default is nicer12:54
dprincespredzy: Yeah. I think I agree12:54
spredzythen I can leave the review as it is right12:54
spredzy?12:54
gfidentebut I don't know if bnemec slagle had other problems which they fixed with the one-per-release element12:55
dprincespredzy, gfidente I'm +2 but will let it sit for a bit12:55
spredzyI am fine with it, let's wait to hear back from bnemec ans slagle12:56
slaglespredzy: commented12:58
*** jprovazn has joined #tripleo12:58
*** rlandy has joined #tripleo12:59
spredzyslagle, thanks13:02
openstackgerritYanis Guenane proposed openstack/tripleo-image-elements: Default to latest rdo-release  https://review.openstack.org/20361113:02
*** regebro has joined #tripleo13:03
*** ukalifon2 has joined #tripleo13:04
*** spzala has joined #tripleo13:04
*** ukalifon has quit IRC13:05
dprincespredzy: 2 nits :)13:05
dprinceshardy: is there a ticket up wtr to "heat stack hanging" issues?13:06
gfidentespredzy, I would also add rdo-kilo :)13:06
spredzydprince, I don't understand : May as well bump this to something recent too no?13:08
spredzyerf13:08
spredzymy bad13:08
spredzyI ll put juno13:08
spredzy:)13:08
dprincespredzy: I just meant 'kilo'13:08
spredzysince it's the current/default one13:08
spredzyI wanted to put something different to highlight it13:08
dprincespredzy: sure13:09
spredzygfidente, should I also add juno then ?13:09
dprincespredzy: either works13:09
gfidentespredzy, ignore me13:10
spredzygfidente, ahaha why ?13:10
gfidentespredzy, I thought icehouse/juno were there13:10
spredzyI am seriously asking :)13:10
spredzyicehouse is juno isn't13:10
gfidentespredzy, if we miss juno it means we prefer setting the VAR13:10
gfidenteand I'm fine with that13:10
spredzyok13:10
openstackgerritYanis Guenane proposed openstack/tripleo-image-elements: Default to latest rdo-release  https://review.openstack.org/20361113:11
shardydprince: Not sure which issue you mean - is it where puppet sometimes quietly gets stuck and the stack times out?13:12
shardydprince: mmagr has a patch which makes that easier to debug : https://review.openstack.org/#/c/188737/13:13
dprinceshardy: the issue I'm seeing is puppet finishes, sends the notification (200 OK is recieved from Heat API), but then no subsequent deployments are recieved13:13
shardydprince: Hmm, I saw that once, but only with network isolation, I assumed it was something related to my environment as I haven't been able to reproduce it13:14
shardydprince: if you do heat resource-metadata <stack> <server id> do you see the pending deployments?13:15
gfidentedprince, shardy me saw that too13:15
shardye.g the nested stack id for e.g OS::TripleO::Controller and the name of the OS::Nova::Server resource13:15
*** adrianopetrich has joined #tripleo13:16
gfidentedprince, shardy I think there is a job where deployments never started13:16
shardygfidente: Hmm, OK.  I'm still mostly doing non-isolation deployments which never IME have this problem13:16
gfidentedprince, shardy http://logs.openstack.org/04/202804/3/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/00c50b7/ FWOW13:16
shardyso maybe some sort of race related to networking?13:16
dprinceshardy: I'm not using networking this morning13:17
*** pradk has joined #tripleo13:17
dprinceshardy: networking could increase the frequency of this happening perhaps13:17
dprinceshardy:  but I don't think it is solely responsible for it13:17
shardydprince: aha, yeah true, just pointing out the pattern I observed13:18
gfidenteshardy, in that CI job the os-collect-config log seems to never get data13:18
gfidentefor all three controllers13:19
shardyit'd be good to distinguisn "occ can't connect to heat" vs "heat doesn't provide metadata to occ"13:19
gfidenteshardy, computes worked13:19
spredzydprince, ping https://bugs.launchpad.net/tripleo/+bug/146423913:21
openstackLaunchpad bug 1464239 in tripleo "mount: special device /dev/sdb does not exist" [Critical,In progress] - Assigned to Dan Prince (dan-prince)13:21
spredzyI've been bitten by this bug over the week end13:21
uvirtbotLaunchpad bug 1464239 in tripleo "mount: special device /dev/sdb does not exist" [Critical,In progress] https://launchpad.net/bugs/146423913:21
spredzywhat is your workaround to deploy successfully an overcloud ?13:22
dprincespredzy: so we don't need the "ephemeral" partition when using puppet13:22
shardyhttp://logs.openstack.org/04/202804/3/check-tripleo/gate-tripleo-ironic-overcloud-f21puppet-ha/00c50b7/logs/overcloud-controller-0_logs/os-collect-config.txt.gz13:22
dprincespredzy: if you are using upstream devtest.sh you can set your ROOT_DISK == NODE_DISK and then it won't create a Nova ephemeral partition and you can ignore this13:22
shardygfidente: that isn't the most informative, unless occ really couldn't retrieve any metadata at all from the API?13:23
spredzydprince, ack thanks for the tip :)13:23
shardyit also doesnt seem to have the heat_local userdata, which I suppose would mean it has no credentials to poll heat13:23
shardyor is that /var/lib/cloud/cfn-init-data ..13:24
dprincespredzy: http://git.openstack.org/cgit/openstack/tripleo-incubator/tree/doc/source/puppet.rst#n8513:24
*** lblanchard has joined #tripleo13:25
shardyOk, that's from "local" not "heat_local"13:25
gfidenteshardy, yeah unfortunately I've only seen it in CI13:28
*** jdob has quit IRC13:28
gfidenteshardy, so we have to resort to exising logs, can't do much troubleshooting13:28
*** jdob has joined #tripleo13:28
*** regebro has quit IRC13:34
*** matbu has joined #tripleo13:38
*** jdob has quit IRC13:39
*** jdob has joined #tripleo13:39
*** regebro has joined #tripleo13:41
*** goneri_ has quit IRC13:44
*** shardy_ has joined #tripleo13:56
*** ukalifon2 has quit IRC13:57
*** shardy has quit IRC13:57
*** goneri_ has joined #tripleo13:58
*** paramite is now known as paramite|afk14:00
*** shardy_ has quit IRC14:01
*** matbu has left #tripleo14:01
*** ukalifon has joined #tripleo14:01
*** shardy has joined #tripleo14:02
*** paramite|afk is now known as paramite14:02
*** paramite is now known as paramite|afk14:03
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Allow enabling debug mode for config management (Puppet)  https://review.openstack.org/20246514:05
lucasagomescan some core of triple-ci please take a look at https://review.openstack.org/#/c/202629/ ?14:07
*** morazi has joined #tripleo14:07
* jistr looks14:08
lucasagomesthis is fixing the tripleo-ci gate for the python-ironicclient project14:08
jistrlucasagomes: ah good, i was just about to ask what drives the change..14:11
jistri see http://logs.openstack.org/85/203585/2/check-tripleo/gate-tripleo-ironic-undercloud-precise-nonha/bf8e969/console.html#_2015-07-20_14_03_10_42814:11
openstackgerritMerged openstack-infra/tripleo-ci: Install the most recent release of pbr on jenkins node  https://review.openstack.org/20262914:12
jistrlucasagomes: ^ rechecking can begin :)14:13
*** pradk has quit IRC14:13
*** bnemec has joined #tripleo14:15
*** paramite|afk is now known as paramite14:23
*** regebro has quit IRC14:26
*** derekh has joined #tripleo14:26
lucasagomesjistr, uhul! thanks!14:29
*** jprovazn has quit IRC14:34
*** spzala_ has joined #tripleo14:35
*** jcoufal_ has joined #tripleo14:38
*** tteggel_ has joined #tripleo14:39
*** tchaypo_ has joined #tripleo14:39
*** Nakato_ has joined #tripleo14:39
*** adrianopetrich has quit IRC14:39
*** davidlenwell_ has joined #tripleo14:39
*** colettecello has joined #tripleo14:40
*** pblaho_ has joined #tripleo14:40
*** ishant has quit IRC14:41
*** tchaypo has quit IRC14:42
*** spzala has quit IRC14:42
*** rlandy has quit IRC14:42
*** weshay has quit IRC14:42
*** Hunner has quit IRC14:42
*** gothicmindfood has quit IRC14:42
*** pblaho has quit IRC14:42
*** gfidente has quit IRC14:42
*** jcoufal has quit IRC14:42
*** persia has quit IRC14:42
*** davidlenwell has quit IRC14:42
*** tteggel has quit IRC14:42
*** Nakato has quit IRC14:42
*** gus has quit IRC14:42
*** persia has joined #tripleo14:42
*** rlandy has joined #tripleo14:42
*** Hunner has joined #tripleo14:42
*** stendulker has joined #tripleo14:42
*** persia has quit IRC14:42
*** persia has joined #tripleo14:42
*** Hunner has quit IRC14:42
*** Hunner has joined #tripleo14:42
*** masco_ has joined #tripleo14:42
*** tchaypo_ is now known as tchaypo14:42
*** gus has joined #tripleo14:42
*** gfidente has joined #tripleo14:43
*** gfidente has joined #tripleo14:43
*** mcornea has quit IRC14:44
*** weshay has joined #tripleo14:56
*** colettecello has quit IRC14:57
*** gothicmindfood has joined #tripleo14:58
*** goneri_ has quit IRC15:02
*** goneri_ has joined #tripleo15:06
*** jistr is now known as jistr|mtg15:13
*** jcoufal_ has quit IRC15:15
*** jayg has quit IRC15:19
*** jayg|g0n3 has joined #tripleo15:20
*** sdake has joined #tripleo15:20
*** jistr|mtg is now known as jistr15:24
*** paramite has quit IRC15:27
*** sdake_ has joined #tripleo15:27
*** rhallisey has quit IRC15:28
*** akrivoka_ has joined #tripleo15:28
*** akrivoka has quit IRC15:29
*** rhallisey has joined #tripleo15:30
*** sdake has quit IRC15:31
*** marzif has joined #tripleo15:35
*** akrivoka_ is now known as akrivoka15:39
*** sdake_ has quit IRC15:47
*** sdake has joined #tripleo15:49
gfidentejistr, spredzy I think I have to use an exec for this https://review.openstack.org/#/c/201796/3/puppet/hieradata/controller.yaml,cm15:49
gfidenteunless you have other ideas?15:49
*** sdake has quit IRC15:51
*** shivrao has joined #tripleo15:54
*** shivrao_ has joined #tripleo15:57
*** dtantsur is now known as dtantsur|afk15:58
*** aufi has quit IRC15:58
*** alop has joined #tripleo15:59
*** shivrao has quit IRC15:59
*** shivrao_ is now known as shivrao15:59
*** yamahata has joined #tripleo16:02
*** bvandenh has quit IRC16:10
*** lazy_prince has joined #tripleo16:12
*** julim has quit IRC16:22
*** jistr has quit IRC16:24
*** rwsu has joined #tripleo16:26
*** trown is now known as trown|lunch16:27
*** yog__ has quit IRC16:32
*** panda has joined #tripleo16:33
*** akrivoka has quit IRC16:34
*** goneri_ has quit IRC16:40
*** shivrao has quit IRC16:45
*** yamahata has quit IRC16:46
*** regebro has joined #tripleo16:46
*** stendulker has quit IRC16:48
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Drive DB initialization via Hiera  https://review.openstack.org/20360616:50
*** derekh has quit IRC17:00
*** shardy_ has joined #tripleo17:01
*** athomas has quit IRC17:02
*** shardy has quit IRC17:02
*** shardy_ has quit IRC17:06
*** shardy has joined #tripleo17:07
*** athomas has joined #tripleo17:08
*** jayg|g0n3 has quit IRC17:08
*** jayg|g0n3 has joined #tripleo17:09
*** jayg|g0n3 is now known as jayg17:09
*** yamahata has joined #tripleo17:10
*** yamahata has quit IRC17:10
*** yamahata has joined #tripleo17:11
*** shivrao has joined #tripleo17:16
*** openstackgerrit has quit IRC17:16
*** openstackgerrit has joined #tripleo17:17
*** lucasagomes is now known as lucas-dinner17:21
edmondkHello, just curious how close this patch is to getting merged: https://review.openstack.org/#/c/185110/ it is for writing out the trusted-host for pypi? I am currently manually patching diskimage-builder with a similar fix until this gets merged.17:25
*** goneri_ has joined #tripleo17:33
*** regebro has quit IRC17:38
*** regebro has joined #tripleo17:38
*** masco_ has quit IRC17:42
gfidenteedmondk, if you could update it adding the LOG.warning message suggested by marios I think we could merge it quickly17:43
*** pino|work has quit IRC17:57
edmondkk17:57
*** jprovazn has joined #tripleo17:59
*** shardy_ has joined #tripleo17:59
*** pcaruana has quit IRC18:00
*** shardy has quit IRC18:01
*** shardy_ has quit IRC18:05
*** trown|lunch is now known as trown18:05
*** shardy has joined #tripleo18:05
*** regebro has quit IRC18:06
*** sthillma has joined #tripleo18:12
*** ukalifon has quit IRC18:16
*** lazy_prince has quit IRC18:18
*** pelix has quit IRC18:31
*** pblaho has joined #tripleo18:34
*** pblaho has quit IRC18:35
*** ukalifon1 has joined #tripleo18:41
*** lucas-dinner has quit IRC18:52
openstackgerritRyan Hefner proposed openstack/tripleo-heat-templates: Fix Puppet Configuring NetApp Cinder Backend  https://review.openstack.org/19362718:54
*** nijaba has quit IRC18:54
*** sthillma has quit IRC18:55
*** sthillma has joined #tripleo18:55
*** adam_g` is now known as adam_g19:01
*** adam_g has quit IRC19:01
*** adam_g has joined #tripleo19:01
*** lucas-dinner has joined #tripleo19:04
*** jprovazn has quit IRC19:05
*** bvandenh has joined #tripleo19:34
*** bvandenh has quit IRC19:43
*** bvandenh has joined #tripleo19:45
*** bvandenh has quit IRC19:51
*** goneri_ has quit IRC19:53
*** goneri_ has joined #tripleo19:58
*** ukalifon1 has quit IRC20:06
*** jtomasek has quit IRC20:10
rhefnerjdob: ping re: a few things: (1) I posted a screencast on cinder patch showing that it works as expected. (2) I have Manila *almost* there. I'm getting a connection refused and it seems like the Manila service is listening on 192.0.2.24, but not .20 or .21 (the other IP addresses on the controller). It seems like a config issue, but I can't for the life of20:10
rhefnerme figure out where20:10
rhefnerhere's the details https://gist.github.com/rhefner1/009f33490c4bd5255fd520:10
rhefner(where 8776 is cinder and 8786 is manila)20:11
rhefnerand it could be something else entirely :)20:12
jdobhrm20:12
jdobrhefner: is the upstream patch up to date?20:12
rhefnerjdob: no I made some changes. give me a sec20:13
openstackgerritRyan Hefner proposed openstack/tripleo-heat-templates: Enable Manila integration  https://review.openstack.org/18813720:13
rhefnerjdob: ^20:13
*** jcoufal has joined #tripleo20:13
jdobalways handy when the bot links it for you :)20:14
rhefnerjdob: there's also os-cloud-config and rdomanager-oscplugin that may be relevant: https://review.openstack.org/#/c/203217/ and https://review.gerrithub.io/#/c/240498/20:15
*** goneri_ has quit IRC20:15
*** Goneri has joined #tripleo20:16
jdobthe plugin looks fine, that's pretty simple20:16
openstackgerritRyan Hefner proposed openstack/os-cloud-config: Enable Configuring Manila Service  https://review.openstack.org/20321720:16
*** paramite has joined #tripleo20:17
*** hewbrocca has quit IRC20:17
jdobrhefner: you're not using the network isolation stuff right? just a flat network?20:17
rhefnerjdob: I am using the VIP stuff, shardy asked me to put that back in20:18
openstackgerritDan Prince proposed openstack/puppet-tripleo: Add package_manifest resource.  https://review.openstack.org/20380320:20
jdobrhefner: weird; does it look like manila is configured correctly on the node?20:22
jdoblike, can you manually dork with it to get it on all 3 IPs?20:22
jdobnothing in here jumps out at me20:22
rhefnerjdob: just saw that it's not configured in haproxy.conf - where do set that? is that controlled in tripleo-heat or somewhere else?20:22
jdobTHT20:22
rhefnerjdob: ah i see it now in THT. let me add that and try20:23
jdobcontroller.yaml20:23
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Ensure present/latest for puppet driven package updates  https://review.openstack.org/19091820:34
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Write package names out to flat files  https://review.openstack.org/19434820:34
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Puppet: wire in tripleo::packages  https://review.openstack.org/19339420:34
*** gfidente has quit IRC20:34
*** gfidente has joined #tripleo20:36
*** paramite has quit IRC20:38
*** jtomasek has joined #tripleo20:42
*** jayg is now known as jayg|g0n320:45
*** bvandenh has joined #tripleo20:45
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Drop pin to install old Fedora keepalived package  https://review.openstack.org/20381420:48
rhefnerjdob: I changed it in controller.yaml, but haproxy.cfg still doesn't have manila. i don't see any other haproxy config in THT - any idea?20:51
*** panda is now known as panda|Zz20:52
*** bvandenh has quit IRC20:54
jdobrhefner: thinking20:57
*** trown is now known as trown|outttypeww20:58
rhefnerjdob: I added an equivalent to this: https://github.com/openstack/tripleo-heat-templates/blob/master/controller.yaml#L761 . Do I need an equivalent to this? https://github.com/openstack/tripleo-heat-templates/blob/master/controller.yaml#L101221:00
rhefnerI haven't really looked at this file since I thought that only controller-puppet.yaml was used. but I don't see any haproxy there21:00
*** morazi has quit IRC21:00
jdobrhefner: to be honest, I'm not sure about the second one, but I would have thought the first one should have worked21:01
jdobrhefner: you might need to ping gfidente tomorrow morning (he's in italy), he's our resident expert on HA21:01
rhefnerjdob: ok I can do that. another separate issue is that when the manila share service starts up, it looks for the neutron admin user/pass and tries to log in. if it can't find it, the service fails to start (see https://gist.github.com/rhefner1/aeeeebbde5bd93d6157e). currently, all the config is right but I'm guessing that the neutron user is not created21:04
rhefneruntil after puppet is finished running.21:04
jdobshit, that's right21:04
jdobthose users aren't created until the post deploy config21:04
jdobwhich is run by the CLI21:05
rhefnerany way to run puppet after *that* to restart the service?21:05
jdobthough i feel like this isnt the first time we've run into this21:05
jdobi dont think we run puppet though21:05
jdobi think we just executed commands21:05
rhefnerno, I had the same thing with the glance image (tried to log into the glance user to insert)21:05
*** Hunner has quit IRC21:05
rhefnerbut right now, the service just sits on the controller node dead. you'd have to manually log in and restart it for manila to work21:05
jdobya, right now the cli just does a bunch of openstack API calls21:06
*** crinkle has quit IRC21:06
*** crinkle_ has joined #tripleo21:07
rhefnerthis sounds like a problem21:07
jdobthis sounds like a new wrinkle we haven't hit before21:07
jdobha, ya21:07
rhefnerhmm...21:08
jdobslagle: you there?21:08
slaglejdob: for you i am21:10
jdobslagle: <insert witty response here>21:10
jdobslagle: have we ever had a situation in overcloud post config where we didn't use an openstack API to do something?21:11
jdobhitting a hiccup with manila; the service wants to start via puppet, but it can't start until the neutron user is in place21:11
jdobso one option would be tying that service starting to post config (kinda ugly) after the users are created21:12
jdobbut i'm not sure how we'd do that without SSHing into the nodes21:12
slaglewell, init-keystone does a ssh to intialize the pki. but that's kind of a one off thing just for boostrapping security21:14
slaglejdob: by neutron user, you mean the one defined in keystone right? not the system account?21:14
rhefnerslagle: correct21:14
slagleis there a reason why?21:15
slaglewhy is manilla special? :)21:15
rhefnerslagle: the manila share service fails to start unless that user is in place since it uses neutron to host shares on the network21:16
*** crinkle_ is now known as crinkle21:17
slagleright, but that happens all over all the openstack services. i guess they just gracefully handle it?21:18
*** rhallisey has quit IRC21:18
slaglejdob: but, to answer the question...there isnt really a clean way to do it right now i can think of21:18
slaglewe want to move the user/endpoint creation into puppet21:18
slagleonce that's done, there could be another step that enables manila21:19
slaglebut really, it sounds like this might be a bug in manila21:19
jdobwell, shit.21:20
rhefnerjdob slagle: is there any downside to setting up keystone users before puppet runs?21:21
*** rhallisey_ has joined #tripleo21:21
rhefnerIIRC that's what devstack does (though I understand that's not a model deployment)21:21
slaglekeystone isn't up before puppet runs21:22
jdobi think he meant during, which goes back to slagle's comment about us wanting to get there, we're just not yet21:22
jdobi'm not sure why it was done this way in the first place21:22
slagleyou could possible hack something into the puppet for manila that creates the neutron user in keystone, assuming keystone is available at that point21:22
slaglelater on i think setup-endpoints will gracefully handle the user already existing21:23
*** Goneri has quit IRC21:23
jdobwe should be able to control it running after keystone using the step stuff21:23
slaglebut...that's pretty hacky21:23
rhefnerslagle: hacky, but that could work21:23
slaglebefore you go too far with that, you might want to verify that user is all that's needed21:24
slaglewhat's going to happen if the endpoint isn't there? (it won't be)21:24
rhefneractually yeah that won't work - i believe that the service pings the neutron api to see what kind of networks are there21:24
jdobrhefner: does the puppet run fail if the service doesn't start?21:26
rhefnerjdob: no it succeeds. the service sits on the controller node dead though21:27
jdobin theory, we could log into the controllers and bounce the service?21:27
rhefnerjdob: yes that's what I've been doing thus far. if you restart the service, everything starts up fine21:27
slaglewhat a systemd Restart= in the service file?21:28
slagle*about a21:28
jdobcan you set a timeout on that?21:29
slaglei think so21:29
jdober, delay21:29
rhefnerslagle: where does the service file exist? in the RPM? or somewhere else?21:29
jdobya, it'd be in the packaging21:30
*** adrianopetrich has joined #tripleo21:30
rhefnerI see it here https://github.com/openstack-packages/cinder21:30
rhefnerrather https://github.com/openstack-packages/manila21:30
*** adrianopetrich has quit IRC21:30
jdobso we'd need a delay and a number of retries21:31
*** bvandenh has joined #tripleo21:31
jdoband then to tell people that once the cloud is created, there may be a time delay before manila is available21:31
jdobi'd almost rather we just doc it as a known issue that you have to log in and start the service21:31
jdoband then prioritize moving the endpoint/user stuff into puppet21:32
*** lucas-dinner has quit IRC21:34
*** Hunner has joined #tripleo21:36
rhefnerjdob: we're still targeted to get this in A1 (assuming we missed the boat for GA in 6 days) - if endpoint/user stuff can be fixed by the september a1 release, then known issue is fine by me.21:36
jdobi'm emailing about that now21:36
gfidenterhefner, I think manila isn't going to use that user until it starts right?21:37
jdobgfidente: right, that's what we're talking about, how to get the user in place before it starts21:37
rhefnergfidente: right, but when puppet configures it, it starts the service21:38
rhefnerthen it fails21:38
gfidenterhefner, so we don't want puppet to start it21:38
gfidenterhefner, pacemaker should do it21:38
gfidenterhefner, but we do users creation in a step which come past services are started21:38
gfidenterhefner, step6 in controller_pacemaker.ppv21:39
gfidentecan you hardcode a username which won't exist when the service starts but will be created during the deployment?21:39
rhefnergfidente: right, so I'm guessing that the service start will still fail unless it happens after the users are created and endpoints are set up21:39
rhefnerthe problem is that it's not just the user, it reaches out to the neutron api to get information21:40
*** rhallisey_ is now known as rhallisey21:40
rhefneryou can argue that manila should gracefully handle this and I'll talk to bswartz (manila PTL) about it21:40
gfidenterhallisey, well we could set a dependency on neutron, but neutron itself won't work until users are up21:40
rhalliseygfidente, did you mean rhefner ?21:41
*** bvandenh has quit IRC21:41
jdobyes :)21:41
rhallisey:) tab complete is op21:41
gfidenterhallisey, I am sorry21:41
rhalliseygfidente, hehe no problem21:41
gfidentejdob, I think the issue with users is actually bigger than we think21:42
gfidentejdob, we probably hit it only now with manila21:42
jdobi agree, this isn't the last time this is going to be a problem21:42
gfidentebut nobody told us services were supposed to handle this gracefully, as rhefner pointed, we were just sort of lucky ;P21:42
rhefnergfidente: as a side note, we approached this issue with puppet-manila trying to insert an image into glance. that didn't work because the users/endpoints weren't set up yet21:42
rhefnerbut luckily that issue can be resolved with a simple bash script before creating a share21:43
jdobgfidente: where is that controller_pacemaker.ppv?21:43
rhefnerthis requires a user to ssh into controller node and restarting the service. not the end of the world, but also not pretty either21:43
gfidentejdob, overcloud_controller_pacemaker.pp21:43
jdobi dont see a step 6 in there21:44
gfidentejdob, step6 ... and 7 https://review.openstack.org/#/c/180566/21:44
gfidenteslagle, ^^21:44
jdoboh21:44
rhefnerI see a ControllerOvercloudServicesDeployment_Step6 in my deployment21:44
gfidenteor 5 and 621:44
gfidentejdob, precisely meant to slowly get rid of postconfig and do keystone population via puppet21:45
jdobya, that came up earlier, starting to move all of this in there21:45
jdoband then we'll have to have a way of tieing into those steps21:46
*** rhallisey has quit IRC21:46
jdobgfidente: i'm missing something, that'll create all of the service users?21:47
jdoband set up the end points?21:47
jdobor am I misreading what you said21:47
gfidentenah only admin for now and the heat_stack user21:47
gfidenteit's meant to start from there21:47
*** rlandy has quit IRC21:48
jdobgotcha, ok21:49
jdobok, so21:49
*** rlandy has joined #tripleo21:49
jdobwe should be able to get this all done by A121:49
jdobif we finish out this patch and give rhefner a way to hook into the step stuff21:49
jdobto make sure the manila service config/start comes after it21:49
*** jcoufal has quit IRC21:50
rhefnerthat can work21:52
jdoblemme ask the guys with a higher pay grade if that works from them21:52
rhefnerjdob: ack21:52
jdobi'll let you know tomorrow since it's like 3am for one of them and the other is on a plane21:52
rhefnersounds god21:52
rhefner*good21:52
jdobgfidente: you still there to answer an ha proxy config for rhefner ?21:53
gfidenteyes21:54
*** alop has quit IRC21:54
rhefnergfidente: I need to add Manila to the haproxy.cfg file. I changed controller.yaml and made a duplicate of this https://github.com/openstack/tripleo-heat-templates/blob/master/controller.yaml#L761 for manila21:55
rhefnerhowever, the controller node's haproxy.cfg doesn't show manila21:56
gfidenterhefner, wait that's old elements code21:56
gfidenteyou have to add it into puppet-tripleo21:56
gfidenteand optionally enable it when manila is deployed21:56
gfidenterhefner, https://github.com/openstack/puppet-tripleo/blob/master/manifests/loadbalancer.pp21:56
rhefnergfidente: ah I thought that controller.yaml was a little fishy, but jdob and I didn't see anything haproxy related in controller-puppet21:57
jdobshit, glad I asked, i didn't realize we had old shit lying around21:57
rhefnerI've never seen this repo before21:57
gfidenteoh it's actually working code deploying the non-puppet jobs21:57
gfidentejust unmaintained :P21:57
gfidentewhich is probably the reason why it continues to work! :)21:58
rhefnergfidente: ok thanks, I'll take a stab at that21:58
*** dprince has quit IRC21:59
rhefnerthanks for the help jdob22:00
jdobnp rhefner22:00
jdobsthillma: can we move that conversation here? i'm already bothering the pacemaker guy, might as well ask him this one too22:01
jdobgfidente: got another issue for you22:01
sthillmagfidente: yeah we were hitting some issue w/ the “pacemaker::constraint::colocation { 'control_vip-with-haproxy’” constraint w/ single controller22:02
gfidentesthillma, what is the issue?22:04
sthillmadeployment was failing, don’t have the logs handy unfortunately, at the time I just thought you guys had moved to the HA config by default and we needed to change it back to single controller mode22:05
sthillmawill work on reproducing it and getting you the exact error message22:05
gfidentesthillma, so it defaults to using pacemaker with single controller, that is correct, but it shouldn't fail22:07
gfidenteif it does it's a bug22:07
gfidentewhat was the reason to think controlvip/haproxy constraint was the root issue?22:08
sthillmathat was the error I saw in /var/log/messages22:08
*** MasterPiece has joined #tripleo22:12
*** jcoufal has joined #tripleo22:13
*** jcoufal has quit IRC22:13
gfidentesthillma, so I don't think I ever saw this same issue before22:16
sthillmagfidente, ok, stack is deploying now w/ pacemaker config so i’ll send you the error log once i hit it again22:20
gfidentesthillma, output from pcs status should tell which actions failed22:20
gfidente(that is as root from one of the controllers)22:21
*** jtomasek has quit IRC22:22
sthillmaJul 20 18:25:14 localhost os-collect-config: ^[[1;31mError: Invalid parameter constraint_params on Pacemaker::Constraint::Base[control_vip-then-haproxy] at /var/lib/heat-config/heat-config-puppet/f824653c-9607-4940-9955-60598f6bc4db.pp:202 on node overcloud-controller-0.localdomain^[[022:27
sthillmaalthough pcs status does show it as up22:27
sthillmabut no services seem to be up22:29
sthillmagfidente, ^ anything else I should check?22:30
*** yamahata has quit IRC22:34
*** gfidente has quit IRC22:35
*** Goneri has joined #tripleo22:45
*** alop has joined #tripleo22:53
*** chlong has quit IRC22:54
*** prad has quit IRC23:09
*** spzala_ has quit IRC23:13
*** spzala has joined #tripleo23:13
*** bitblt has joined #tripleo23:20
*** yuanying has joined #tripleo23:29
*** spzala_ has joined #tripleo23:37
*** spzala has quit IRC23:38
*** spzala_ is now known as spzala23:39
*** Goneri has quit IRC23:40

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!