Friday, 2016-08-12

openstackgerritPaul Belanger proposed openstack/python-tripleoclient: Initial support for bindep
*** Ryjedo_ has joined #tripleo00:19
*** Ryjedo has quit IRC00:20
*** Ryjedo_ is now known as Ryjedo00:20
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_subnet_map output
pabelangerEmilienM: bnemec: fixes a typo with an ovb job. It should be in experimental-tripleo, not experimental00:23
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_uri_map output
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove unneeded net_ip_map dump into hieradata
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass ServiceNetMap to services
*** limao has joined #tripleo00:28
*** limao has quit IRC00:29
*** limao has joined #tripleo00:29
*** rajinir has quit IRC00:35
*** sarath has quit IRC00:47
EmilienMpabelanger: ack00:49
pabelangerEmilienM: success!
pabelangerovercloud-full job working in experimental00:52
EmilienMpabelanger: awesome :D00:52
*** davidlenwell has quit IRC00:53
pabelangertakes 25mins to build00:53
*** TicToc has joined #tripleo00:59
TicTocbash localhost always ends up with a ssh to the undercloud error even though I can manually ssh into the undercloud instance .. any tips on getting things working01:05
*** davidlenwell has joined #tripleo01:05
*** ChanServ sets mode: +v davidlenwell01:05
*** xuao has joined #tripleo01:06
*** TicToc has quit IRC01:15
*** rhallisey has joined #tripleo01:24
*** zeroshft has joined #tripleo01:26
*** zeroshft has quit IRC01:30
*** limao_ has joined #tripleo01:36
*** limao has quit IRC01:39
*** bana_k has quit IRC01:40
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass ServiceNetMap to services
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Add temporary pin to last known good current tripleo repo
*** rhallisey has quit IRC02:01
*** yamahata has quit IRC02:02
slagleCI failing due to | TripleO : | | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt02:03
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Undecided,New]02:03
*** slagle changes topic to "CI failing due to | TripleO : | | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt"02:03
EmilienMslagle: hey02:03
EmilienMI think I know why02:03
EmilienMisn't it required to the cinder volume thing?02:04
EmilienMcinder volume type ing*02:04
slagleyea we promoted a bad repo accidentally02:04
slagleit's in the bug02:04
EmilienMgfidente has a patch iirc02:04
gfidenteoh god guys02:04
gfidenteI abandoned those all02:05
EmilienMslagle: what's the fix now?02:05
gfidentethey fixed it in cinder02:05
EmilienMgfidente: you here????02:05
gfidenteand that's where the fix belongs :)02:05
EmilienMgfidente: it landed 53 minutes ago02:05
slagleEmilienM: it's in the bug02:05
gfidentezaneb, is teaching me some python on a heat submission02:05
EmilienMswift also broke02:06
slagleEmilienM: we need to get the rdo admins to restore the previosu symlink, or we can go with this pin,
EmilienMbut I have a patch
EmilienMslagle: dmsimard is here02:06
EmilienMdmsimard: can you help dude?02:06
dmsimardhelp what02:06
slagledmsimard: check your mail02:07
EmilienMcan you restore the previous tripleo symlink please?02:07
gfidenteslagle, shouldn't we get a new promote now that the fix is landed in cinder?02:07
*** yamahata has joined #tripleo02:07
slaglei emailed it to you02:07
slaglegfidente: no. b/c of like 2 or 3 new unrelated issues02:07
slaglethe swift thing being one of them02:07
EmilienMslagle: fixed by
EmilienMslagle: see REGEX skills dude02:08
dmsimardslagle: the buildlogs content is a CDN02:08
dmsimardupdating the symlink will not be effective immediate02:08
dmsimardbut I can check if the symlink was updated properly02:08
gfidenteEmilienM, right, I believe without asking02:08
dmsimardslagle: the dlrn server has current-tripleo -> c6/bd/c6bd3cb95b9819c03345f50bf2812227e81314ab_4e6dfa3c right now02:09
dmsimardyou want 8e/b0/8eb0c893074f6ba912114ce50995ee0118930775_fc22fbaf/ ?02:09
EmilienMdmsimard: yeah, should be ok for us02:10
*** ooolpbot has joined #tripleo02:11
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]02:11
*** ooolpbot has quit IRC02:11
gfidenteI am leaving guys, drank too much02:11
slagledmsimard: yes, we need to go back to the 8e/b0 one02:11
gfidentebetter to smoke now02:11
EmilienMgfidente: lol02:11
gfidentehave good night02:11
*** gfidente has quit IRC02:11
EmilienMslagle: do you want me to +2 +A your ooo-ci patch?02:12
slagleEmilienM: if you want. we won't need it if the symlink gets updated02:12
dmsimardslagle: done lrwxrwxrwx.   1 root          root              55 Aug 12 02:12 current-tripleo -> 8e/b0/8eb0c893074f6ba912114ce50995ee0118930775_fc22fbaf02:12
EmilienMso we don't need the ooo-ci patch02:12
EmilienMdmsimard: thx dude02:13
dmsimardslagle: please note that buildlogs is not updated immediately02:13
slagledmsimard: thanks02:13
dmsimardI will reply to the email02:13
EmilienMdmsimard: ok so Swift should be fixed in my patch02:13
EmilienMdmsimard: Nova error looked like random, since it passed on scenario00102:13
EmilienMwe should be good for a promotion this time...02:13
*** slagle changes topic to "TripleO : | | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt"02:15
dmsimardafk now02:15
slagleme 202:15
slaglenight folks02:15
EmilienMnight folks02:15
*** zeroshft has joined #tripleo02:27
*** bswartz has quit IRC02:33
*** zeroshft has quit IRC02:36
*** bswartz has joined #tripleo02:37
*** zeroshft has joined #tripleo02:42
*** tzumainn has quit IRC02:46
*** zeroshft has quit IRC02:46
*** ayoung has joined #tripleo02:49
*** rhefner has quit IRC02:55
*** rhefner has joined #tripleo02:55
*** mwhahaha has quit IRC02:55
*** mwhahaha has joined #tripleo02:59
*** Ryjedo has quit IRC03:01
*** zeroshft has joined #tripleo03:02
*** zeroshft has quit IRC03:03
*** Ryjedo has joined #tripleo03:03
*** zeroshft has joined #tripleo03:03
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]03:10
*** zeroshft has quit IRC03:12
*** limao_ has quit IRC03:13
*** yamahata has quit IRC03:14
*** links has joined #tripleo03:33
*** limao has joined #tripleo03:38
*** ayoung has quit IRC03:45
*** numans has joined #tripleo03:46
*** ramishra_ has joined #tripleo03:49
*** ramishra has quit IRC03:49
*** liverpooler has quit IRC03:49
*** masco has joined #tripleo04:02
*** numans has quit IRC04:03
*** ooolpbot has joined #tripleo04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]04:10
*** fragatina has quit IRC04:16
*** limao has quit IRC04:19
*** limao has joined #tripleo04:20
*** fragatina has joined #tripleo04:58
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port
*** skramaja has joined #tripleo05:00
*** radeks has joined #tripleo05:02
*** fragatina has quit IRC05:03
*** jaosorior has joined #tripleo05:04
*** ooolpbot has joined #tripleo05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]05:10
*** TicToc has joined #tripleo05:19
TicTocstill looking for some help on the Localhost install and the ssh into undercloud or some other tips I can try05:23
jaosoriorTicToc: It's pretty early at the moment for people. Might need to wait for later when more people log in05:27
jaosoriorI'd say a couple of hours05:27
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Include journal in the outputted logs
*** coolsvap has joined #tripleo05:34
*** r-mibu_ has quit IRC05:46
*** r-mibu has joined #tripleo05:46
*** rcernin has quit IRC05:51
*** hewbrocca-afk is now known as hewbrocca05:53
*** saneax_AFK is now known as saneax05:54
*** saneax is now known as Guest8973905:54
jaosoriorhewbrocca: hey man, do you remember what happened to that cinder volume type issue that was blocking RDO promotino?06:00
*** xuao has quit IRC06:01
openstackgerritMerged openstack/tripleo-common: Fix upload-swift-artifacts with keystone v3
hewbroccajaosorior: I think we worked around it in t-h-t, but then something else (of course) came along that is blocking rdo promotion06:07
*** limao has quit IRC06:08
*** limao_ has joined #tripleo06:08
*** ooolpbot has joined #tripleo06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]06:10
*** jprovazn has joined #tripleo06:11
jaosoriorhewbrocca: I think this bug is related to the volume type issue ^^06:12
*** limao_ has quit IRC06:15
*** limao has joined #tripleo06:15
jaosoriorah, funky, multinode-nonha seems to be passing now06:16
*** fultonj has quit IRC06:17
*** rcernin has joined #tripleo06:22
*** aufi has joined #tripleo06:23
*** pgadiya has joined #tripleo06:30
*** pgadiya has quit IRC06:31
hewbroccajaosorior: huh does that mean we can close that bug?06:38
jaosoriorhewbrocca: lets wait a few more patches to pass CI06:40
ccamachoHappy Friday guys!06:47
Jokke_stormy Friday it is06:50
ccamachoHere in Madrid its like to be in the Sahara :P06:52
Jokke_Got one of those "Unplug everything" Thunderstorms coming over abount an hour ago06:54
bandinimorning folks06:56
bandiniccamacho: lovely 16 degrees here :P06:56
* bandini *ducks*06:56
bandinifolks, can we get some reviews on the monitoring agents reviews: CI passed, I tested it as well and we need it for our upcoming opstools work.06:57
bandinieternal glory and beers from me as reward :)06:58
*** tesseract- has joined #tripleo06:59
ccamacho:) ajaj  bandini, deploying your submission, after it finishes, what can I run to check that actually works as  expected?07:02
*** ebalduf has quit IRC07:03
*** florianf has joined #tripleo07:06
*** ooolpbot has joined #tripleo07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]07:10
bandiniccamacho: so just set some bogus monitoring parameters ( and verify that in /etc/sensu/*json everything is configured correctly07:11
bandiniccamacho: a more thorough test needs the monitoring node stuff to land, and that is not finished yet07:11
ccamachonice :) Ill let you know when it finishes07:12
bandiniccamacho: thank you, sir ;)07:13
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: Include journal in the outputted logs
*** yamahata has joined #tripleo07:19
*** jpena|off is now known as jpena07:25
*** akuznetsov has joined #tripleo07:31
*** ebarrera has joined #tripleo07:38
*** abehl has joined #tripleo07:40
*** abehl has quit IRC07:41
*** iogg has joined #tripleo07:42
*** akuznetsov has quit IRC07:43
*** pgadiya has joined #tripleo07:44
*** amoralej|off is now known as amoralej07:44
*** Guest89739 has quit IRC07:45
*** ifarkas_afk is now known as ifarkas07:46
*** saneax has joined #tripleo07:46
*** saneax is now known as Guest3408507:47
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond
*** jpich has joined #tripleo07:49
*** ohamada has joined #tripleo07:49
*** xuao has joined #tripleo07:49
jaosoriorhewbrocca, shardy, I guess we can close that bug 1612434. Doesn't seem to be failing anymore07:49
openstackbug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]
*** iogg has quit IRC07:49
*** milan has joined #tripleo07:50
*** xuao has quit IRC07:51
*** xuao has joined #tripleo07:51
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bond
openstackgerritSaravanan KR proposed openstack/os-net-config: Add support for OVS DPDK Bridge and Port
*** hewbrocca is now known as hewbrocca-afk07:54
*** yamahata has quit IRC07:55
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Availability monitoring agents support
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Monitoring node support
mariosbandini: heh we pushed the button at the same time08:00
mariosbandini: ah ok my vote wasn't wiped so np08:00
bandinimarios: yeah it was just a rebase, although I did explicitly add --no-rebase to git review so I have no idea how that happened08:01
mariosbandini: well, was gfidente anywhere near you when this occurred?08:01
bandinimarios: ahahahah08:01
*** Goneri has joined #tripleo08:04
*** athomas has joined #tripleo08:07
*** social has joined #tripleo08:08
*** TicToc has quit IRC08:08
*** ooolpbot has joined #tripleo08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]08:10
*** shardy has joined #tripleo08:11
Jokke_shardy: morning08:13
shardyJokke_: Hi!08:13
Jokke_shardy: I think Won't Fix is the right one to close that
openstackLaunchpad bug 1589922 in tripleo "OSD data dir expected in /var/lib/ceph/osd" [High,In progress] - Assigned to Erno Kuvaja (jokke)08:14
*** xuao has quit IRC08:14
shardyJokke_: ack - gfidente said he was looking into a fix yesterday, I assume you guys have discussed it?08:14
Jokke_shardy: actually we haven't ... that whole thing was deemed too dagerous to fix due to upgrades08:15
*** lucas-dinner is now known as lucasagomes08:15
Jokke_that was at least the last discussion we had with wider audience08:16
*** xuao has joined #tripleo08:16
Jokke_and it's annoyance at max anyways08:16
shardyJokke_: Ok, sounds good, well lets sync with gfidente when he comes online and if you guys agree we'll wontfix it and remove from n-308:16
shardythanks for the update08:16
shardyI agree it seems risky for something pretty minor08:16
*** skramaja has quit IRC08:18
*** dbecker has joined #tripleo08:20
*** stendulker has joined #tripleo08:25
*** TicToc has joined #tripleo08:26
*** TicToc has quit IRC08:27
*** xuao has quit IRC08:28
*** electrofelix has joined #tripleo08:28
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Allow to skip kernel cleanup
*** hewbrocca-afk is now known as hewbrocca08:29
*** bfournie1 has joined #tripleo08:31
*** bfournie has quit IRC08:33
*** bfournie1 has quit IRC08:36
*** gfidente has joined #tripleo08:37
*** gfidente has quit IRC08:37
*** gfidente has joined #tripleo08:37
*** numans has joined #tripleo08:37
*** Goneri has quit IRC08:38
*** anshul has joined #tripleo08:43
*** radeks has quit IRC08:46
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Use non-bracketed version of addresses for keystone.initialize
ccamachobandini my local env hangs on any deployment..  still trying to make it work..08:48
*** oshvartz has joined #tripleo08:49
shardyAnyone looked into why even the nonha job is now taking over 2.5hrs to run?08:51
shardysome jobs are getting killed by the infra timeout even when everything has worked, it's just taking too long :(08:51
shardyMy local testing indicates we're spending a really long time in Step3 because of keystone stuff08:52
*** dtantsur|afk is now known as dtantsur08:52
*** d0ugal has quit IRC08:52
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass ServiceNetMap to services
*** Goneri has joined #tripleo08:53
bandiniccamacho: oh ouchers08:55
ccamachoshardy that might the same issue hitting me today08:55
ccamacho**might be08:56
ccamachoI'm waiting for the deployment to fail and check what's going up there08:57
bandiniI did not notice that last night, will retry this morning and see what's up08:57
shardygfidente: Hey, re passing the ServiceNetMap - I had one alternative idea08:57
shardywhat if we wrote the ServiceNetMap and the per-node NetIpMap in hiera08:58
*** athomas has quit IRC08:58
shardythen used a two step hiera interpolation?08:58
shardye.g like that08:58
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements
dtantsurhey folks! I've got a new bunch of changes for ironic-in-overcloud up:
dtantsurthis brings us to the point of nova-compute installed on the controller and recognizing Ironic resources08:58
dtantsurfolks keen on using THT for installing undercloud should definitely review it ;)08:58
shardygfidente: then in the service template, it'd just be like "%{hiera(\"HeatApiNetwork\")}"08:58
dtantsurthe WIP docs if you want to try it:
gfidenteshardy, right but didn't we want to do it in heat?08:59
gfidentewhat I was doing is setting bind_host: {get_param: ServiceNeMap}09:00
shardygfidente: sure, either way is valid - I just thought previously we said the hiera approach was not possible because interpolation can't do a map lookup09:00
gfidenteao that map_replace would replace internal_api with the net_ip_map address09:00
shardythis would be a way around that09:00
gfidenteif we wanted to lookup say the _uri or _subnet09:01
jaosoriorshardy: What keystone stuff is taking that long?09:01
gfidentewe'd have to do the mangling in heat anyway to make hiera interpolation to work09:01
*** numans has quit IRC09:01
shardyjaosorior: creating roles, users and stuff09:01
shardyit takes forever09:01
shardyI think retry logic was added which means it's spinning on each step09:02
shardychem: ^^ can you confirm, puppet-keystone now retries all-the-things because keystone was being flaky?09:02
jpichIs tripleo-common part of the "non-client library freeze" or is it something else? Would a cycle-trailing deadline also apply here?09:02
shardygfidente: Yeah, true, Ok cool, just wanted to mention it :)09:02
gfidentethe hiera way tough doesn't require us to pass servicenetmap to the services09:02
*** athomas has joined #tripleo09:03
gfidentebut I like heat way of doing09:03
shardygfidente: ack, fine we can continue with the heat approach for sure09:03
shardygfidente: got me thinking about the hiera options, that's all :)09:04
*** xuao has joined #tripleo09:04
shardyI was working on that yesterday with EmilienM, so we can adjust the puppet providers to enable interpolating the per-service rabbit_ips09:04
hewbroccaoh, ugh, is that why we're running out of memory setting up keystone?09:07
hewbroccaIt seems absurd that it should be this difficult...09:07
shardyhewbrocca: Yeah, we need to investigate further, it shouldn't be this difficult, and we shouldn't really need while(true) around openstackclient stuff in puppet09:08
marioshewbrocca: is it multicore boxes? because puppet-keystone default makes for bad performace (but is/should be fixed in master) sec there was a bz09:08
shardysomething in either our ordering, or keystone, must be broken09:08
jaosoriorthis looks suspicious in puppet-keystone
marios shardy is this somethign in ci you are talking about /me reads back09:10 bug 1347305 in openstack-tripleo-heat-templates "Overcloud deployed with keystone as single process leads to abysmal performance" [Unspecified,On_qa] - Assigned to mandreou09:10
*** ooolpbot has joined #tripleo09:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]09:10
jaosoriorspecially that last sentence in the commit message09:10
jaosoriorshardy ^^09:11
*** panda|zz is now known as panda09:12
*** pkovar has joined #tripleo09:13
*** numans has joined #tripleo09:13
shardyKeystoneWorkers: 1 in CI09:13
*** chem has quit IRC09:15
shardyOk lets try increasing that, but we've also got OOM issues, so probably not by too much09:15
mariosshardy: well there is also a lp for it fwiw
openstackLaunchpad bug 1598092 in tripleo "Performance issues with keystone under wsgi apache on a multinode machine" [Medium,Fix released] - Assigned to Marios Andreou (marios-b)09:15
shardymarios: ya, but I think with your patch landed, we still force the number of workers to 1 in CI09:15
gfidentemarios, shardy though I think we're running with 1 worker from quite a while now09:16
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Try increasing KeystoneWorkers to improve performance
shardylets see how that goes09:18
*** limao has quit IRC09:18
gfidenteok I'm looking at ordering09:18
mariosshardy: ack at least it is still overriding the  Threads to 1 (in hiera) so possibly it isn't the same issue. I mean, bug was was because workers 1 and processor threads but the ci env had then 1 worker (override) and 1 thread (we specify via hiera for that bugfix)09:18
shardymarios: ya, but with that change to KeystoneWorkers, and your patch that landed, we'll increase keystone::wsgi::apache::workers from 1, right?09:19
shardyand the bz says to increase that while leaving threads as 1, so hopefully the CI change may help?09:19
mariosshardy: yeah if you > in ci the override it should help09:20
shardycool, thanks, lets see how long it takes09:20
* shardy will time it locally too09:20
shardyAnother wierd thing is keystone-manage bootstrap seems to get stuck sometimes09:21
shardyit's possible that's related to my local environment though, but it doesn't fail gracefully when keystone isn't responding09:22
shardyjust gets wedged forever09:22
*** chem has joined #tripleo09:23
jaosoriorshardy: chem just logged in ^^09:24
chemjaosorior: hi, well, I just reconnect actualy :)09:25
openstackgerritGabriele Cerami proposed openstack/tripleo-quickstart: Devmode: rename delorean current so any existing repo is overwritten
gfidentethough looking at our keystone profile is substantially unchanged from june09:26
jaosoriorchem: aah, alright, sorry about the extra ping then09:26
chemjaosorior: but I did lost some minutes of log, what's the matter ?09:26
shardychem: Hey, I noticed that our deployment step where we configure keystone is taking a long time recently09:27
*** stendulker has quit IRC09:27
shardychem: I wanted to confirm, are there retry loops around things like creating roles and users via openstackclient in puppet-keystone now?09:27
gfidentebut I think one thing we could try is to move ::roles and ::endpoint in 409:27
gfidenteso I'll try that09:27
chemshardy: they have always been retry around *all* openstack call in puppet lib.  The default timeout/retry has indeed been modified some time ago: each call is tried during a longer time and the overall time we retry stuff as been increased as well09:29
chemshardy: been done to accomodate CI09:29
shardychem: Ok, thanks09:29
openstackgerritGiulio Fidente proposed openstack/puppet-tripleo: Move ::roles and ::endpoint after ::keystone and ::wsgi
gfidente^^ that's an attempt to avoid the retries09:32
shardychem: is there a similar timeout around the /usr/bin/keystone-manage bootstrap command?09:33
chemshardy: hum ... unlikely. let me check.09:33
shardychem: In my local testing it looks like it's possible for that to get stuck (if keystone is misconfigured or otherwise unresponsive)09:34
*** jpeeler has quit IRC09:35
chemshardy: keystone-manage is simply a exec puppet resource, no provider, no default retry.09:36
shardychem: Ok, thanks, that explains what I was seeing then09:36
*** mgould|afk is now known as mgould09:47
*** jpeeler has joined #tripleo09:50
*** ooolpbot has joined #tripleo10:11
*** ooolpbot has quit IRC10:11
openstackLaunchpad bug 1612434 in tripleo "multinode jobs failing during pingtest with "Resource CREATE failed: ResourceInError: resources.volume1: Went to status error due to "Unknown""" [Critical,In progress]10:11
*** Ryjedo_ has joined #tripleo10:11
*** Ryjedo has quit IRC10:12
*** Ryjedo_ is now known as Ryjedo10:12
*** coolsvap is now known as _coolsvap_10:13
openstackgerritMartin André proposed openstack/puppet-tripleo: UI profile for tripleo
*** tosky has joined #tripleo10:15
openstackgerritMartin André proposed openstack/instack-undercloud: Introduce 'enable_ui' option
*** dtantsur is now known as dtantsur|brb10:20
* rook reads back.. shardy you guys are going to fix the KeyStone worker bug where you create 1 process and # of cores as threads? -- there are other services that are like that not sure if you want to tackle those too10:26
*** xuao has quit IRC10:27
*** leanderthal|afk is now known as leanderthal10:27
*** xuao has joined #tripleo10:29
shardyrook: Yeah, but we'll have to look at where the fix goes - traditionally the defaults to launch number of workers == number of cores happened in oslo, but that doesn't apply with the move to httpd for many services10:32
rookright, it is a total regression wrt Performance.10:32
rookwrt Keystone and Ceilometer for example10:32
shardyrook: Sure, but we've generally just deployed the service defaults, so it'd be nice if there was a common solution which just made those defaults sane again10:33
shardyvs a special-case for TripleO10:33
shardyif we need to it can be calculated and set via puppet tho10:33
rookshardy: we have _tried_ to fix this upstream just get the whole: We don't tune for performance bullshit answer.10:33
rookand it isn't a performance thing, it is just a regression10:34
rookwe even shared performance results.10:34
shardyrook: Ok, well that definitely sucks, if folks want to httpd all the things they should deal with the fallout IMO10:34
*** pkovar has quit IRC10:34
shardyrook: raise a tripleo bug if you haven't already (performance results would be great), we'll figure out some way to wire in sane defaults10:35
rookwe have one for keystone for sure.10:35
shardyrook: we can also discuss with EmilienM, sounds like something the puppet community may already be seeking to address in a common way10:37
*** xuao has quit IRC10:38
*** dsariel has quit IRC10:43
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: [WIP] Parameters listing and updating
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Allow to skip kernel cleanup
*** dsariel has joined #tripleo10:55
openstackgerritMerged openstack/tripleo-heat-templates: Reinstate CloudDomain properties for Controller/Compute roles
slagleanyone looking into the latest CI error? Exception registering nodes: Node 59ecc491-41b0-4950-8d00-ddce532ba0f4 is locked by host localhost.localdomain, please retry after the current operation is completed11:09
shardyslagle: I just saw that one a moment ago, until then most failures I've seen have been timeouts11:10
jistrchem: hi, are you still looking into the puppet-pacemaker refactoring? is that something likely to land in Newton? (The reason i ask is that i'm kinda tempted to make an attempt to solve our 'services are always restarted on stack-update and scale up' problem. The solution would probably be slightly different based on if we do or do not have the refactoring already present.)11:11
shardyhappended here, not yet looked into why or reproduced11:11
slagleshardy: ok it's affecting the periodic job as well, wasn't sure about check-tripleo11:11
openstackLaunchpad bug 1612622 in tripleo "CI: Exception registering nodes: Node 59ecc491-41b0-4950-8d00-ddce532ba0f4 is locked by host localhost.localdomain, please retry after the current operation is completed" [Critical,In progress] - Assigned to James Slagle (james-slagle)11:12
slaglebug filed anyway11:12
shardyslagle: I pushed because many patches I looked at have hit the infra timeout even for the nonha job11:12
shardyslagle: my local testing indicates that is at least in part because the keystone performance during step3 is horrible11:13
chemjistr: basically everything is in place.  I was waiting for the pacemaker light architecture (pla) to land to make the refactoring on puppet-triplo/tht .  I don't know the current status of the pla, do you ?11:14
slagleshardy: yea i've noticed the same11:14
*** weshay has joined #tripleo11:14
chemjistr: when newtown is supposed to land ?11:14
jistrchem: if you mean the next gen architecture where we don't manage OS services with pacemaker, i think that's already landed now11:15
jistrchem: IIRC the code freeze for Newton is around the end of this month or beginning of next one11:15
*** yolanda has quit IRC11:16
shardywe agreed to observe feature freeze at the same time as other projects, so the end of this month, with the possibility of feature freeze exceptions11:17
chemoki, so newton will be with the new pacemker architecture.  If it has landed, then, I could tell you by next mid-week if there is hope to have it landed before the end of the month.  Would it be ok for you ?11:17
*** thrash|g0ne is now known as thrash11:18
*** yolanda has joined #tripleo11:18
chemjistr: ^ we could arrange a quick meeting to discuss its state, WDYT ?11:18
jistrchem: yea sure. Just to clarify the intention -- you want to arrange the meeting next week or now? I'm fine with either.11:19
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: Only set  if it is not already defined
jistrchem: possibly next week we'll have more data though. I could *hopefully* (just got pinged with 2 BZs :D) look into possible approach re the restart problem and try guesssing how much it's interwoven with the puppet-pacemaker refactoring. E.g. can we do it in parallel, or do we have to land the refactoring first.11:23
jistrchem: i think the landcape changed quite a bit with the next gen HA for both (both the refactoring and the restart issue)11:23
jistrchanged for the better i should say :)11:24
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: update service workers to match upstream
chemjistr: on Wednesday if that's fine by you. I can arrange it. It gives me time I have to have a look at the current state of the tht and my patch refactoring (Monday is off in France)11:28
jistrchem: sure, WFM, thanks :)11:28
*** jaosorior has quit IRC11:29
chemjistr: ack, nice.  Thanks for reaching out. I'll send the invitatation as soon as I've deciphered how to do such thing with google stuff.11:30
*** jaosorior has joined #tripleo11:30
*** maeca1 has joined #tripleo11:32
jaosoriorwhois weshay11:32
jaosoriorweshay, larsks, are you guys around?11:32
jaosoriorweshay: hey dude, quickstart related question11:33
jaosoriorweshay: So...I'm deploying my overcloud nodes with network isolation. And I see that the network-environment.yaml is autogenerated. Where is the DnsServers option coming from?11:33
jaosoriorsomehow when I have network isolation I'm not able to ping it. Thought it would be the hypervisor host there11:34
*** dtantsur|brb is now known as dtantsur11:34
weshayjaosorior, undercloud/defaults line 2311:35
weshaythe external network is defined in roles/common/defaults/main.yml line 9111:36
jaosoriorweshay: alright, makes sense... now it seems there's a route missing or something of the sort there. The one that's configured by default by quickstart is not usable if I use network isolation11:38
jaosoriorEmilienM: hey dude, how's it going?11:38
*** jistr has quit IRC11:39
EmilienMjaosorior: good thx11:41
Jokke_jaosorior: I would have assumed _isolation_ being indication for that, maybe jut me ;)11:41
*** StevenK has quit IRC11:41
*** StevenK has joined #tripleo11:42
jaosoriorJokke_: the nameserver is in the management network... not sure if the nodes not being able to access the name sever which is in the management network is assumed with network isolation11:42
gfidenteshardy, jaosorior in a combination of rechecks this passed all jobs
gfidenteI would actually merge it before we need more rebases and such11:46
*** masco has quit IRC11:52
Jokke_jaosorior: I might have been trolling just a tiny bit with that comment ;)11:55
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: update undercloud & multinode job names
EmilienMslagle: ^11:56
jaosoriorEmilienM: hey dude, what's your opinion on ? I actually now unsure if we need ~>11:56
openstackgerritMarios Andreou proposed openstack/tripleo-common: Minor update to the
slagleEmilienM: thanks. i was wondering what was up with that :)11:56
jaosoriorJokke_: hah fair enough. I need coffee11:57
EmilienMjaosorior: ok keep it11:57
EmilienMjaosorior: +211:57
EmilienMshardy, rook: what's up?11:58
jaosoriorAny +A for this?
*** Guest34085 is now known as saneax_AFK12:00
*** lucasagomes is now known as lucas-hungry12:00
jaosoriorthanks slagle12:01
*** ccamacho is now known as ccamacho|lunch12:02
*** morazi has joined #tripleo12:04
shardygfidente: ack, yes, thanks for approving! :)12:05
*** jayg|g0n3 is now known as jayg12:06
shardyEmilienM: Hey, there has been discussion re the default workers/threads for services that are moving to httpd, e.g keystone12:06
shardypreviously oslo.service defaults mostly meant that it was easy to default to workers == num_cpu's12:06
shardybut now the defaults generally appear to be 1, which results in a performance regression on upgrade12:07
EmilienMshardy: afik we didn't change the default in puppet-*12:07
EmilienMit must be in THT services12:07
*** pkovar has joined #tripleo12:07
*** bfournie has joined #tripleo12:07
EmilienMoh wait12:07
EmilienMI am wrong12:07
shardyEmilienM: THT doesn't know the number of CPUs per node, can we detect that through some puppet facts?12:07
EmilienMsure we can12:08
EmilienMso let me rephrase my statement :12:08
EmilienMhttpd config in puppet-* modules are using 1 by default (I don't know why)12:08
EmilienMexample here:
shardyEmilienM: Ok, well started this discussion12:08
*** morazi has quit IRC12:08 bug 1347305 in openstack-tripleo-heat-templates "Overcloud deployed with keystone as single process leads to abysmal performance" [Unspecified,On_qa] - Assigned to mandreou12:08
shardyit appears that for upgrading operators, that is a bad/surprising new default12:08
EmilienMso yeah, we could override the value12:09
EmilienMand use the fact12:09
shardywhen previously taking no action resulted in num_cores12:09
*** morazi has joined #tripleo12:09
EmilienMdo we want to use $::processorcount ?12:09
shardyif that's not going to be handled in the puppet modules, I guess we'll have to change THT for every service that's under httpd12:09
EmilienMworkers = $::processorcount ?12:09
shardyEmilienM: Yeah, but we'll still need to allow override via the *Workers parameters12:09
EmilienMwe can change it in puppet modules12:10
shardyEmilienM: anyway, not something we have to do right now, I just wondered if the puppet community were already addressing it, e.g by setting defaults to processorcount12:10
EmilienMthe question is to what12:10
EmilienMwe set: threads to $::processorcount already12:10
EmilienMdo we would need to set $workers too?12:10
ansiwenhi EmilienM12:10
shardyEmilienM: ah, well in the bz it suggests threads=1 and workers=processorcount12:11
EmilienMwe have physicalprocessorcount and processorcount facts12:11
shardyso we may need to do some testing and see if those settings are interchangeable12:11
EmilienMok so thread would be physicalprocessorcount12:11
*** iogg has joined #tripleo12:11
EmilienMand workers=processorcount12:11
ansiwenregarding the buckets_path problem of the puppet-ec2api: it's set here
shardyEmilienM: Yeah, what is suggested in the bz is workers=processorcount and threads=112:12
*** bfournie has quit IRC12:12
shardyI *think* that is the closest we can get the old behavior, but I've not tested to compare threads/workers count12:12
EmilienMthreads=1 ? we need to hardcode 1?12:12
ansiwenEmilienM: does it mean the default value is wrong, or do we always have to set it accordingly when instantiation ec2api::s3 ?12:12
*** ohamada has quit IRC12:13
EmilienMsomeone tried already12:13
shardyEmilienM: aha, that is the reporter of the BZ ;)12:14
EmilienMshardy: I'll take over the patch12:14
EmilienMand try again12:14
shardyEmilienM: thanks - we may need some more performance results to establish good defaults I guess12:14
shardyrook: ^^ can you help with that?12:14
*** rhallisey has joined #tripleo12:15
shardyEmilienM: I can say the current settings cause horrible performance for keystone, I've seen it locally12:15
EmilienMshardy: how urgent is it?12:15
*** ohamada has joined #tripleo12:15
shardyEmilienM: Not super urgent, but something we need to fix before users upgrade to newton12:15
EmilienMshardy: ok I'll fix it today if I can12:16
EmilienMtrying to prioritize a bit :)12:16
shardyEmilienM: sure, I'd leave it until next week if you have other stuff to do, just wanted to discuss it :)12:16
EmilienMansiwen: I haven't looked depper but it looks like the default value in ec2api might be risky or packaging broken. Again I have no idea about ec2api, I tried to help you on the puppet side12:17
*** amoralej is now known as amoralej|lunch12:23
*** jistr has joined #tripleo12:24
*** jpena is now known as jpena|lunch12:25
pabelangerpanda: myoung: So, this passed last night: It is the experimental job I added to build overcloud-full.qcow2 outside tripleo-ci12:26
shardyEmilienM: thanks! :)12:27
*** fultonj has joined #tripleo12:27
ansiwenEmilienM: where does the packaging happen? is that dlrn? or rdo?12:29
EmilienMdlrn is rdo12:30
*** jistr has quit IRC12:31
*** jistr has joined #tripleo12:32
*** chlong|mtg has quit IRC12:33
ansiwenEmilienM: I thought dlrn is built for fedora on a nightly basis, and RDO is build for centos releases?12:35
*** fultonj has quit IRC12:35
*** fultonj has joined #tripleo12:36
*** dbecker has quit IRC12:37
*** jistr has quit IRC12:41
EmilienMansiwen: no, dlrn builds at each commit in OpenStack, and we support Fedora and CentOS.12:42
EmilienMplease read
pandapabelanger: we're still low on undercloud images .. :) can you show me the source of this job ?12:42
pabelangerpanda: JJB config:
pabelangerpanda: does python-tripleoclient build undercloud images today?12:45
*** rlandy has joined #tripleo12:45
pabelangerpanda: or what is the syntax to build undercloud images12:46
pandapabelanger: no12:46
slaglewe should not be using python-tripleoclient to prebuild uc images12:47
slaglewe shouldnt be using prebuilt uc images at all12:47
pabelangerslagle: how do undercloud images get built?12:47
pabelangeror is that an operator choice?12:47
slaglepabelanger: you run the undercloud installer on a centos host12:48
slaglethere aren't any predelivered images12:48
pabelangerslagle: because I don't know, what is the undercloud installer?12:50
slagle"openstack undercloud install"12:50
*** anshul has quit IRC12:51
*** bfournie has joined #tripleo12:51
pabelangerokay, so that expects your OS to already be installed12:52
larsksOn my compute node I have hiera_include('compute_classes', []) in /var/lib/heat-config/..., but on my controller I have hiera_include('controller_classes') (note the lack of a default), which is failing.  Why is the hiera_include call different on the controller?12:52
*** dbecker has joined #tripleo12:52
pabelangerslagle: so, that said, why was tripleo-ci building undercloud images before ovb?12:52
larsksAnd is the fact that controller_classes doesn't seem to be defined anywhere indicative of a problem, or is that typical?12:53
*** links has quit IRC12:53
*** tzumainn has joined #tripleo12:53
slaglepabelanger: for quickstart aiui12:54
shardylarsks: that appears to be a bug that is fixed on master12:56
slaglepabelanger: i think we also cached them. but only if it has passed tripleo-ci12:56
*** jdob has quit IRC12:56
*** jdob has joined #tripleo12:57
larsksshardy: actually, that's what was puzzling me, because I'm running from master.  But it seems something is still picking up the installation in /usr/share/openstack-tripleo-heat-templates, even though I am passing "--templates tripleo-heat-templates" to overcloud deploy.12:57
slaglepabelanger: i'd like to clarify how we are going to get these images validated by tripleo-ci before we move forward adding more builds for quickstart to consume12:57
shardylarsks: Are you updating an existing deployment?12:57
slaglei dont want to end up with 2 different processes12:57
larsksshardy: In what sense?  I am deploying a new overcloud (not updating an existing one).12:58
larsksI am going to try removing the directory from /usr/share and see what happens.12:58
shardylarsks: if you're updating an existing overcloud we do a PATCH update, so --templates foo isn't enough to update all the content inside heat12:59
larsksYeah, not updating an existing overcloud.12:59
shardyfor a new deployment, it shouldn't be using anything from /usr/share when you pass --templates12:59
*** amoralej|lunch is now known as amoralej12:59
larsksI will see if I can reproduce this while paying attention to things.12:59
shardylarsks: ack - FWIW I'd probably run the deploy with --debug & tee the result so you can be absolutely sure what got passed into heat in the files map13:00
larsksWill do.13:00
*** lucas-hungry is now known as lucasagomes13:00
pabelangerslagle: I don't see quickstart consuming these images, I think that happens some place else.  Basically, this image is the tip of master for all dependent projects.   However validation of that image happens in the next step in the pipeline.  But for now, we are just confirming python-tripleoclient and build it13:01
larsksshardy: actually, take a look at this:
larsksGiven '--templates tripleo-heat-templates', why is it looking in /usr/share/openstack-tripleo-heat-templates?  Am I making a stupid cli error?13:02
shardylarsks: you're passing a bunch of environments from usr/share13:02
larsksOh, duh.13:02
openstackgerritMartin André proposed openstack/puppet-tripleo: UI profile for tripleo
shardyyou probably need to rewrite the prefix for all of them13:02
larsksSorry, that's what I get for using a script :/13:02
* larsks runs and hides.13:02
*** lblanchard has joined #tripleo13:03
*** jistr has joined #tripleo13:03
*** pkovar has quit IRC13:04
slaglepabelanger: as long as what we validate builds as tripleo-ci does13:04
slaglenot using pip, git, etc13:04
bswartzmarios: ping13:08
mariosbswartz: hi13:09
gfidenteshardy, marios this actually passed CI and moved ::roles a bit later13:12
gfidentebut doesn't look like it's making deployment any faster?13:12
*** jistr has quit IRC13:13
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: M/N upgrade fix galera restart.
bswartzmarios: hey thanks for starting work on I'm going to take a look at that and offer feedback13:13
*** fragatina has joined #tripleo13:13
bswartzmarios: did you do any refactoring of the manila generic driver code such that it only installs with m-shr (and not m-sch and m-api)?13:13
*** ccamacho|lunch is now known as ccamacho13:14
*** jistr1 has joined #tripleo13:14
shardygfidente: Yeah, and it actually doesn't look like step3 is the slowest thing there either13:15
*** jcoufal has joined #tripleo13:15
*** tbarron|gone is now known as tbarron13:16
shardyWe are spending nearly an hour running puppet on the controller in total though13:16
gfidentestep4 1264.013:17
*** fragatina has quit IRC13:18
*** dprince has joined #tripleo13:18
mariosbswartz: hey np, i just had a first pass yesterday to see what it looked like. you were right about the rebase. I didn't do any refactoring to any of the generic code yet. but i was mainly just aiming to tidy up a bit puppet-tripleo side where we are unnecessarily grabbit hiera data for things that are actually set correctly in the templates manila service files13:19
*** dprince has quit IRC13:19
mariosbswartz: there is a question on the review about which backends we enable by default, and whether or not it makes sense to have more than one - it could affect how the review shapes up13:19
gfidenteshardy, interestingly step5 taking almost as much even though it's not doing much13:20
*** [1]cdearborn has joined #tripleo13:20
shardygfidente: Yeah, that is strange, that seems to run quite fast locally13:20
*** dprince has joined #tripleo13:20
shardyI guess we'll have to profile each step and figure out where the time is going13:20
bswartzmarios: yes, that brings up a related question13:21
bswartzmarios: I'm trying to test tripleo and I don't know where to configure the list of enabled services13:21
bswartzor enabled drivers13:21
mariosbswartz: you mean to enable manila generic for example?13:21
bswartzmarios: both to enable the manila service at all, and which manila driver to enable13:22
mariosbswartz: so this is the example review we used for testing
bswartzokay but how would a user do it?13:22
bswartzhow to end users configure their tripleo cloud how they want?13:22
mariosbswartz: for manila generic, you should just have to enable the manila-generic-config.yaml env file...13:23
bswartzhow do you "enable" an env file13:23
mariosbswartz: and similarly if you also wanted netapp, the way the review is right now, you'd also include -e13:23
mariosbswartz: ah sorry...13:23
mariosbswartz: i mean you include that environment file in your deployment command13:23
*** akuznetsov has joined #tripleo13:23
mariosbswartz: so openstack overcloud deploy --control-scale .... etc -e /environments/manila-generic-config.yaml13:24
openstackgerritMerged openstack/tripleo-heat-templates: Convert EndpointMap to not require per-service VIP parameters
bswartzso regarding your earlier question, we should enable some manila driver by default13:24
bswartzgeneric would be acceptable but honestly we should add support for lvm (another open source driver which is much simpler) and use that as the default13:25
tbarronbswartz: we should prob get lvm plugin in for that13:25
tbarronthere are distribution issues with generic that we can talk about13:25
bswartztbarron: +113:25
bswartztbarron: indeed13:25
tbarroneither that, or don't introduce any env by default but require a choice when manila is deployed13:26
mariosbswartz: so the way the review is right now, we'd include both manila-generic.config.yaml and also the manila-netapp-config.yaml for getting both backends... but that then brings us back to the question on the review13:26
bswartzmarios: sounds like tbarron says yes, and I concur13:27
*** jistr1 is now known as jistr13:27
mariosbswartz: ack, if you could please add a comment/thoughts there then I will udpate, probably Monday at this point13:27
bswartztbarron: you could add more color, which would help:
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: use overcloud_templates_path for all template references
tbarronbswartz: marios bbiam, in a bluejeans meeting13:28
*** akuznetsov has quit IRC13:28
*** jpena|lunch is now known as jpena13:31
jaosoriorAny +A for this?
openstackgerritMerged openstack/instack-undercloud: Add constraints needed for ironic and inspector
tbarronbswartz: marios i will update the review later today, am a bit overcommitted this morning, but ...13:33
tbarronbswartz: marios we will need to split generic off as its own env parallel to netapp, cephfsnative, etc.13:33
tbarronbswartz: marios i am going to study what cinder does w.r.t. lvm, whether is is set up by default, etc.13:34
tbarronbswartz: marios or whether you just choose which envs you want to deploy and none are on by default13:34
tbarronbswartz: marios Jokke_ i was talking with Jokke_ yesterday on this13:35
rbradyjtomasek: can I find out more information about your environment?  I'm not able to replicate the bug you mentioned in the comments on
tbarronbswartz: marios Jokke_ it may be good not to have any enabled_backends or enabled_protocols on by default, i.e. these comma-separated-string values would be constructed dependinng on which envs are deployed13:36
tbarronbswartz: marios Jokke_ we need to compare with cinder volume and the newly deployed cinder backup for this question as well13:37
rbradyjtomasek: I tried to replicate what you pasted and created a new default plan, added a param, changed a param and added additional params.
* jtomasek looks13:37
jtomasekrbrady: yeah, and now run the gerParameters action13:38
mariosbswartz: OK, sounds good and lets use the review for discussion. WRT 'split generic off' ... we could restructure the files, right now generic and netapp config all goes into manila-base.yaml. Wether or not each is then activated will depend on the inclusion or ommission of the relevant environment file (assuming we go with default everything off)13:38
jtomasekrbrady: that's what fails13:38
mariostbarron: err tbarron sorry ^13:38
rbradyjtomasek: ahh...trying now13:39
*** rodrigods has quit IRC13:39
*** rodrigods has joined #tripleo13:40
tbarronmarios: ack, will study it, that's helpful13:40
mariostbarron: i am personally fine with having the config for all the backends in one place, we could move into driver specific files too13:40
mariostbarron: but that is orthogonal to whether or not a backend is enabled, is my point13:40
Jokke_tbarron: yeah, that all will need to wait (at least on my part) for next week13:40
tbarronmarios: sure, i don't have any religion on that, care more about the behavior13:40
openstackgerritRich Bowen proposed openstack/tripleo-quickstart: rst format, rather than markdown.
tbarronJokke_: your weekend and marios weekend should be starting now!13:41
*** trozet has quit IRC13:41
mariostbarron: hahaha wishful thinking13:42
* marios cries13:42
tbarronmarios: :(13:42
Jokke_tbarron: agreed13:43
rlandydmsimard: I dropped the flavor requirements for ovb - to match ben's flavors as closely as possible. I have successful runs on mitaka - I am still debugging master13:45
rlandywe would have to just try a few runs on your cloud.13:45
rlandyI don't have access to create exact flavors in central-ci13:45
dmsimardrlandy: there's no heat yet though and I won't have the time to do it for a while .. although the deployment recipes for that cloud are open source and could have someone help :)13:46
rlandydmsimard: I am on the hook to document baremetal in general first - after that I can try13:47
dmsimardrlandy: ok13:47
*** trozet has joined #tripleo13:50
*** d0ugal has joined #tripleo13:51
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove unneeded net_ip_map dump into hieradata
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_subnet_map output
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Remove deprecated net_ip_uri_map output
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Pass ServiceNetMap to services
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates: Convert ServiceNetMap to a nested template
EmilienMbnemec: to get a tripleo ci promotion, doing recheck on would be enough?13:56
EmilienMI always forget13:56
*** pkovar has joined #tripleo13:56
EmilienMbnemec: modulo L161 that we might need to clean13:57
jaosoriorgfidente: +A for this ? :D13:57
radezdprince: ping, this was +2ed a while back but never merged:
radezdprince: is there something missing for it to be mereged?13:58
gfidentewe need a button in gerrit to post a message on the irc channel when something is ready13:59
mandreEmilienM: thanks for the review of
mandreEmilienM: so jsut to make sure I understand correctly, only the (currently) commented code will make it to this patch13:59
*** myoung is now known as myoung|mtg13:59
mandreEmilienM: the rest will live in THT, correct?14:00
slaglebandini: is httpd still managed by pacemaker?14:00
bandinislagle: nope14:00
dprinceradez: looking14:00
EmilienMmandre: in fact, the class you created will never exist, you need to move bits to other services/profiles14:00
dprinceslagle: can you review this for radez
dprinceslagle: easy one14:01
dprinceradez: 2 +2's are required14:01
dprinceradez: then we +A it14:01
radezdprince: ah, right on, thx dprince slagle14:01
EmilienMI approved it, lgtm14:02
mandreEmilienM: ugh... so I guess we don't need a UI profile if it's all spread out to different services/profiles anyway14:02
shardyslagle: Hey, if I need to change the name of OS::TripleO::Services::NeutronServer, is there any way I can do it now the multinode job is gating?14:02
shardyI can't think of a way because of the reference to the name in tripleo-ci for the all-in-one services list14:03
shardyI guess I'll have to leave it until we move that environment into tht?14:03
*** pkovar has quit IRC14:03
EmilienMshardy: do you remember how we can manually trigger the promotion job ?14:03
EmilienMshardy: is it
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Remove httpd from
dprinceshardy: can't you just add a new key with the same name/effect?14:03
shardydprince: aha, yeah that will work14:04
shardythanks :)14:04
slaglebandini: can u review
*** pradk has quit IRC14:05
slaglei got ping'd by 3 people in the last 3 minutes :)14:05
bswartzslagle: ping14:05
bandinislagle: ping14:05
* bandini chuckles14:05
bandinislagle: looking14:06
hewbroccaslagle: ping14:06
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job
EmilienMslagle: ^14:06
slaglei just hope hewbrocca doesnt ping me. i wouldnt see it though since he is /ignored14:06
* hewbrocca cries14:07
bnemecEmilienM: I have very little idea how promotion works.14:07
slagleEmilienM: please kill that asap14:07
*** bnemec is now known as beekneemech14:07
EmilienMbnemec: me too, but I'll try to get it today14:07
EmilienMslagle: kill what14:07
slagleEmilienM: it will false promote14:07
slaglesee the bug i filed yesterday14:07
*** pradk has joined #tripleo14:07
slagleEmilienM: the overriding of JOB_NAME means that if mitaka passes, it will report the master job as passed to the promote script14:08
EmilienMjobs are killed, it's good.14:08
slagleEmilienM: that's why we had to rollback the symlink yesterday :)14:08
EmilienMnote: when you abandon/restore a patch, it kills CI jobs14:08
EmilienMslagle: ok so how can we get promotion?14:08
slagleEmilienM: look at how i'm doing it here:
slaglei think it is safe14:09
EmilienMI did recheck14:10
EmilienMslagle: but you didn't export the jobtype14:10
EmilienMdoes it matter to actually promote?14:10
slagleEmilienM: promote is triggered by CACHEUPLOAD14:11
slaglewhich i explicitly set14:11
slagleEmilienM: please review the patch :)14:11
slaglei think it's safe, but i can't be online today until 10pm to make sure in case it goes awry :)14:11
slagleit's my "don't attempt CI trickery as friday afternoon approaches" rule14:12
EmilienMhopefully we can promote before14:12
EmilienMI see 0 blocker now14:12
EmilienMpuppet ci promoted 1h ago14:12
*** ebalduf has joined #tripleo14:13
slagleEmilienM: yea, i just got a successful ha deploy locally using consistent14:13
slagletrying pingtest now14:13
*** _coolsvap_ is now known as coolsvap14:16
honzadprince: hey, was the tripleo deep dive recorded yesterday?  do you happen to have a link?14:16
dprincehonza: I think slagle recorded it, slagle got a link yet?14:18
slaglehonza: all the recording links are in the etherpad14:18
hewbroccadprince: what sort of propaganda did you dispense yesterday?14:19
dprincehewbrocca: radical ideas man14:19
honzaslagle: awesome, thanks14:19
dprincehewbrocca: pretty standard network deep dive14:19
openstackgerritMerged openstack-infra/tripleo-ci: update undercloud & multinode job names
openstackgerritRich Bowen proposed openstack/tripleo-quickstart: rst format, rather than markdown. Change-Id: Ib50837db5f85cca117684949299069843242873f
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Update for new HA arch
slaglebandini: updated based on your comments ^14:23
slaglebandini: do we need to add some sort of wait now though?14:23
slagleb/c all the calls to check_resource are removed14:23
bandinislagle: good point. we definitely were not entirely clean there previously either (i.e. we did not wait fully for all services to be up, just openstack-core)14:25
gfidenteso I don't want to14:26
gfidentebut I think and are delicate14:26
gfidenteand good to merge quickly, shardy, EmilienM, jaosorior ^^14:26
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move Nova settings out of puppet/compute.yaml
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Mv Ceilometer settings out of puppet/compute.yaml
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: Move neutron settings out of puppet/compute.yaml
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: use overcloud_templates_path for all template references
gfidentejaosorior++ :)14:27
dprincejaosorior, EmilienM: ^^ more composable work to cleanup compute.yaml14:27
* dprince is working on controller.yaml now14:27
gfidentedprince++ :)14:28
EmilienMdprince: did you sync with jaosorior who is also doing the same thing iirc?14:28
*** saneax_AFK is now known as saneax14:28
*** saneax is now known as Guest4756514:28
dprinceEmilienM: well I pointed out the BZ's which I filed14:28
jaosoriorgfidente: seems to me that this one hasn't passed nonha14:28
dprinceEmilienM: I thought we agreed to track the rest of the composable work in LP bugs (not BZs)14:29
shardygfidente: thanks, be good to see those land :)14:29
EmilienMdprince: cool I'm just saying that because I saw patches from jaosorior14:29
openstackLaunchpad bug 1604412 in tripleo "Composable services: move hiera settings out of compute role" [High,In progress] - Assigned to Dan Prince (dan-prince)14:30
dprincejaosorior, EmilienM: sorry, I thought we discussed this 2 weeks back14:30
dprinceEmilienM: this is why I liked my etherpad14:31
jaosoriorEmilienM, dprince it's all good. I just did some parameters but it's no biggie14:31
jaosoriorI actually did them cause I needed to fix something else. And then found those accidentally14:31
jaosoriorwhen I submitted those patches I actually wasn't aware of the LP14:31
dprinceshardy: are you actually working on this, because I'd like to do it next
openstackLaunchpad bug 1604414 in tripleo "Composable services: move hiera data out of controller role" [High,Triaged] - Assigned to Steven Hardy (shardy)14:31
jaosoriorand then EmilienM pointed it out14:31
EmilienMno biggies! it's friday14:32
shardydprince: Not got to it, feel free to take it :)14:32
larsksshardy: should referencing a missing parameter in the output section cause a failure?14:32
shardydprince: been busy trying to battle through the custom-roles refactoring14:32
dprinceshardy: I just did the same for compute.yaml which besides a few local bind ports is very clean here now:
shardydprince: Nice, thanks, will check it out14:33
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't deploy net-iso env when we won't use it
shardylarsks: it should, but there's a heat bug which means we're not strictly validating outputs right now14:33
dprinceshardy: right, so you are planning on doing this then
*** paramite has quit IRC14:33
dprinceshardy: FWIW, this is directly related to the roles work in that roles won't work without this14:33
shardy larsks14:33
openstackLaunchpad bug 1599114 in heat "Outputs aren't correctly validated" [Medium,In progress] - Assigned to Oleksii Chuprykov (ochuprykov)14:33
shardydprince: I'm not planning to do the controller refactor atm so please feel free to take it14:34
dprinceshardy: for example I tried, MySQL. The composable service is incomplete because the controller.yaml still has many of the MySQL settings in it still14:34
beaglesci is still sad?14:34
shardydprince: I'm refactoring allNodesConfig primarily14:34
larsksshardy: thanks.  I spent hours (hours!) yesterday tracking down a problem that was actually a missing parameter in a deeply nested template.14:34
*** myoung|mtg is now known as myoung14:34
dprinceshardy: ack, cool. Appreciate the work on all nodes14:35
larsksThe error ultimately produced by heat was unhelpful.  I will take a look at the bug.14:35
dprinceshardy: is the local bind IP prototype close yet?14:35
shardydprince: gfidente has some patches which I think he refreshed today14:35
shardydprince: the map_replace part for all the roles has landed14:35
shardylarsks: Yeah, it's definitely easy to be misled because the error isn't surfaced clearly14:36
gfidentedprince, hey TIP is, when that passes we can start moving things14:36
dprincegfidente: boom, thanks14:36
shardywe need that heat bug fixed, there is a patch bug I've not had time to test it14:36
shardylarsks: if you felt like trying it and providing feedback, that would be helpful14:36
gfidenteshardy, actually I think I also got the repeat/for_each change to use maps to pass14:37
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-quickstart: use overcloud_templates_path for all template references
*** jistr has quit IRC14:38
*** jistr has joined #tripleo14:39
shardygfidente: nice!  That will be super helpful14:39
gfidenteI had a use case14:39
gfidentewas curious to dive a little into heat14:40
*** pkovar has joined #tripleo14:40
gfidentezaneb helped14:40
shardyCool, will try to find time to review/test later14:40
* zaneb +2s14:42
*** pkovar has quit IRC14:44
openstackgerritLars Kellogg-Stedman proposed openstack/puppet-tripleo: [WIP] updates to fluentd support
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: [WIP] Add fluentd service
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-heat-templates: [WIP] Add fluentd client service
larsksIf someone has a moment to look at I would appreciate feedback as to whether or not that is heading in the right direction.14:51
slagleEmilienM: well the promote may or may not happen14:53
slagleEmilienM: in my local env, node registration is hanging14:54
*** rajinir has joined #tripleo14:58
*** Guest47565 is now known as saneax_AFK15:05
openstackgerritJohn Browning proposed openstack/tripleo-validations: Added check for ceilometerdb size
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart: Only set $LOCATION if it is not already defined
*** leanderthal is now known as leanderthal|afk15:09
jaosoriorslagle: What's your output for $ mistral action-get baremetal_introspection.wait_for_finish15:10
*** ayoung has joined #tripleo15:10
jaosoriorslagle: if the inputs for that action are empty you have to re-run the mistral-db-manage populate command15:13
jaosoriorand it was an ordering issue in the undercloud15:13
*** toymachine has joined #tripleo15:13
slaglejaosorior: it finally finished but took 20 minutes15:13
jaosoriorah, ok, then it was probably not that then15:13
*** ebarrera has quit IRC15:13
slaglepossibly related to the poor keystone performance we've been seeing?15:13
jaosoriorslagle: that's for the overcloud15:14
jaosoriorthe undercloud keystone should be alright15:14
jaosoriorslagle: well, strange still15:14
jaosoriorAnyway, I gotta go15:14
jaosoriorhave a good weekend folks!15:15
*** jaosorior has quit IRC15:15
slagleshould it be processes=1 threads=4 ?15:15
*** hewbrocca is now known as hewbrocca-afk15:15
slaglei only have a single keystone-main process15:15
*** bana_k has joined #tripleo15:15
slagleshardy: do we need to increase processes or threads for the keystone wsgi conf?15:16
*** yamahata has joined #tripleo15:20
shardyslagle: according to its the workers, which I think KeystoneWorkers will do ref bug 1347305 in openstack-tripleo-heat-templates "Overcloud deployed with keystone as single process leads to abysmal performance" [Unspecified,On_qa] - Assigned to mandreou15:23
*** florianf has quit IRC15:23
*** pcaruana has quit IRC15:23
*** jistr is now known as jistr|call15:23
shardyslagle: I've not done comparitive testing on performance tho, just going by those reports15:23
shardythe tripleo-ci patch seems inconclusive15:23
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart: fix documentation directory in the readme file
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Decouple EndpointMap from SSL certificate params
slagleshardy: thanks15:28
slaglewill play with it a bit15:28
shardyslagle: ack, thanks15:28
shardyI think we need to profile the step4/5 stuff too, as it seems to actually be worse than step3 in CI, despite my impressions from local testing15:28
*** beagles is now known as beagles_brb15:29
slaglei guess i'd expect them to be worse15:30
*** florianf has joined #tripleo15:31
*** ifarkas is now known as ifarkas_afk15:33
*** ayoung has quit IRC15:33
dmsimardEmilienM: what do you think of doing the same "client-side" promotion for tripleo ci repository than is done in p-o-i ?15:35
EmilienMdmsimard: ++15:35
dmsimardEmilienM: i.e, instead of nightly periodic job that promotes if successful, we do the promotion client side in tripleo15:35
EmilienMit would avoid us to promote manually15:35
EmilienMdmsimard: thought it could be done in RDO CI, no?15:36
EmilienMso we can run it more often15:36
EmilienMevery 24h is not enough IMHO15:36
dmsimardEmilienM: we can run it as often as required, just need to submit new patchsets15:36
dmsimardThere'd be a proposal bot review every 24h, but that doesn't prevent us from submitting reviews on specific hashes whenever required15:36
slaglewhy would we move tripleo ci promotion to rdo ci?15:37
dmsimardslagle: that's not what I'm saying15:37
EmilienMslagle: we're talking about puppet ci15:37
dmsimardslagle: puppet-openstack-integration has it's own self-contained promotion mechanism
dmsimardslagle: i.e, it doesn't rely on a symlink server side, it carries the exact hash in it's manifest and updates it as required (automatically or manually)15:38
EmilienMit would be great to run a 6h periodic job in RDO CI that does puppet promotion for us15:38
dmsimardEmilienM: we could just submit a new patchset to p-o-i every 6h :p15:38
EmilienMit sucks15:38
slagleok we used to do that in tripleo-ci as well15:38
EmilienMit still requires people to +2 +A the patch15:38
slaglei dont really remember why we switched to the symlink15:38
EmilienMwe want something automtic15:38
EmilienMwhen I leave in PTO nobody takes care of puppet CI promotion15:39
EmilienMwhile symlink, it would be automatic15:39
EmilienMoh I have an idea15:39
slaglei think it was b/c the CI and docs would get out of sync15:39
EmilienMand if the RDO job fail, submit the change in Puppet CI15:39
EmilienMso it's visible15:39
*** aufi has quit IRC15:39
dmsimardEmilienM: it'd hardly be automatic, it stills needs to be troubleshooted if it fails15:39
EmilienMuntil it fails; promote every 6h; done15:40
EmilienMright !15:40
*** numans has quit IRC15:40
dmsimardEmilienM: and it probably has less visibility in rdo ci than it does in upstream15:40
EmilienMbut if it fails, submit it upstream15:40
EmilienMso we can see the patch and we know it fails15:40
EmilienMsubmit a patch "Failed to promote RDO trunk" in poi upstream15:40
EmilienMwith a RDO gerrit account15:40
EmilienMit would be awesome15:40
dmsimardI guess15:41
*** jistr|call is now known as jistr15:42
EmilienMdmsimard: openstack infra wouldn't let us to run 6h periodic jobs15:44
dmsimardEmilienM: I mean a periodic job on our end15:44
dmsimardthat would create a patch every 6h15:44
dmsimardit's cheating15:44
*** jpich has quit IRC15:45
EmilienMdmsimard: no, I mean, do the promotion (with symblink) ever 6h in RDO CI15:45
EmilienMdmsimard: and if the promotion job fails, submit a patch in p-o-i with the hash15:45
EmilienMso we can see the output in upstream jobs15:46
EmilienMand maybe send only 1 patch? or use git amend?15:46
*** rcernin has quit IRC15:47
openstackgerritMerged openstack/tripleo-heat-templates: Move gnocchi's kestone::auth parameters to API profile
openstackgerritMerged openstack/os-cloud-config: Adding the congress service
dmsimardEmilienM: yes, I understand what you mean15:49
dmsimardEmilienM: I just ideally don't want to offload upstream work down to RDO15:49
dmsimardwe help upstream a lot but we don't want to carry that responsability15:49
dmsimardsubmitting a patch in case of failure is fine with me15:49
jristslagle: ping, re: mistral stuff15:51
jristslagle: we've got a boatload of things that need to land in mistral/cli for the UI for Newton N-315:51
*** weshay is now known as weshay_brb15:51
jristslagle: rbrady has kindly detailed a bunch of them here15:51
openstackgerritMiles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package
*** bana_k has quit IRC15:51
jristslagle: additionally, d0ugal is working on a patch for the deployment via CLI here:
d0ugaljrist, slagle: FYI, I have added it to the TripleO meeting agenda for Tuesday to try and get more eyes.15:52
jristd0ugal: well, with respect, I'm hoping to get more eyes sooner15:53
jristthan Tuesday15:53
jristd0ugal: but I appreciate that :)15:53
d0ugaljrist: aye, that'd be nice but it isn't that long away either :) Just letting you know15:53
slaglejrist: are blueprints filed?15:54
jristslagle: with regard to what? the UI needs or the CLI/Mistral needs?15:54
d0ugalslagle: it is mostly based on the Mistral spec15:54
jristslagle: and that is kind of a general question15:54
jristslagle: so I am not sure if I can answer it to your satisfaction15:55
slaglethat's the remaining work15:55
slagle(so i thought)15:55
*** TicToc has joined #tripleo15:55
slagleare we talking about new things? or things already known?15:55
jristthat's the definition of work needed15:55
thrashd0ugal: sucks that the swift client in OSC is lacking. :/15:55
jristno new things15:55
jristslagle: that i know of15:55
jristat the moment15:55
d0ugalthrash: lol, yeah, it's weird and really limited15:55
jristslagle: but we need this dozen or so patches to land for both the UI, and the CLI15:56
d0ugalthrash: I think they are doing that in an effort to reduce deps15:56
jristslagle: I'm starting to get a little concerned15:56
thrashd0ugal: that'd be my guess15:56
openstackgerritMerged openstack/tripleo-heat-templates: Align node_ips hiera keys with the service name.
thrashjrist: I'll start taking a look as soon as I get a clean env going.15:57
*** beagles_brb is now known as beagles15:57
slaglejrist: yea, understood15:57
slaglei'm not aware of any spare capacity15:58
jristthat's fair15:58
jristso would you say you have a similar level of concern?15:58
slaglejrist: i don't anticipate being 100% feature complete on a lot of things by newton-315:58
slagleso, yes, if that's what you mean15:59
d0ugalBreaking news :)15:59
jristslagle: do you think it'd be worth cutting anything... and is there anything to cut15:59
jristor should we wait a few days to see if we can land some stuff15:59
slagleif the main features are landed, we can always follow up with bugfixes15:59
*** weshay_brb is now known as weshay15:59
slagleif the features are not landed, we can propose them as exceptions15:59
slagleand i think all these critical blueprints would likely be approved exceptions16:00
slaglejrist: there is nothing to cut that i'm aware of16:00
slaglebut these things tend to change the closer we get to deadlines16:00
*** bana_k has joined #tripleo16:00
jristslagle: thanks for the explanation16:02
*** yamahata has quit IRC16:03
*** dsneddon has quit IRC16:08
*** dsneddon has joined #tripleo16:09
ansiwenEmilienM: looks better now:
EmilienMansiwen: +216:13
ansiwenEmilienM: oh, you can do that although you are the owner?16:14
EmilienMyou're the last commiter16:14
EmilienMit's fine in this case...16:14
ccamachoGuys quick question.. just have finished to install manually OOO from master, following the docs, now the AODH profiles were not included in puppet-tripleo returning a Could not find class tripleo::profile::base::aodh::api for overcloud-co... error   Do you know if there are any issues with the packages from RDO?16:16
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate the deploy command to use the Mistral workflows
ccamachoseems to be an old version16:16
slagleccamacho: what version do you have?16:17
honzajrist: this deep dive recording about networking is pretty much perfect, thanks16:17
ccamachoI deleted the env, but re doing all to give you the version.16:18
ccamachoslagle ^16:18
*** florianf has quit IRC16:18
ccamachowill take about 30 mins16:18
slaglecheck the repos that are used for the oc image building16:19
jristhonza: :)16:19
ccamachoslagle, these  and
ccamachobut 2 days ago worked fine, the issue comes from yesterday or today I believe16:21
ccamachoslagle, yeahp aodh its not there16:23
slagleccamacho: that repo is 9 days old16:24
slagleccamacho: are you also using tripleo-heat-templates from that same repo?16:25
ccamachoThe base URL points to but in that RPM the AODH profiles are not there16:25
slagleif you're using tht latest, with that repo, i'd execpt there might be some incompabilities16:25
*** yamahata has joined #tripleo16:26
openstackgerritRyan Brady proposed openstack/tripleo-common: Adds parameters actions
ccamachoThats the issue THT its from master16:27
rbradyjtomasek: I've added the ResetParams action you asked for the morning.  Please see the test.  I'm still working on the heatclient validation issue.
openstackgerritBen Nemec proposed openstack/tripleo-docs: Document decoupled SSL config
*** fragatina has joined #tripleo16:30
*** fragatin_ has joined #tripleo16:31
*** athomas has quit IRC16:32
openstackgerritGiulio Fidente proposed openstack/python-tripleoclient: Use non-bracketed version of addresses for keystone.initialize
*** fragatina has quit IRC16:35
*** ohamada has quit IRC16:35
*** tesseract- has quit IRC16:35
openstackgerritDan Sneddon proposed openstack/tripleo-specs: Spec: undercloud-ntp-server - Add NTP services to Undercloud
jtomasekrbrady: thanks!16:36
*** electrofelix has quit IRC16:37
*** dtantsur is now known as dtantsur|afk16:41
openstackgerritHaïkel Guémar proposed openstack/tripleo-ui: Add packaging spec and config files
*** [1]cdearborn has quit IRC16:49
gfidenteshardy, you still around? regarding I don't think we can append a char to the default format or we'd go back to the upgrade/rename problem16:50
openstackLaunchpad bug 1600209 in tripleo "HostnameMap can result in partial replacement" [High,Triaged] - Assigned to Steven Hardy (shardy)16:50
gfidenteshardy, had other ideas for a THT fix?16:50
openstackgerritMerged openstack/tripleo-heat-templates: Add parameter for enabling purging configuration files
openstackgerritAlan Bishop proposed openstack/os-net-config: Enhance and fix NIC mapping feature
*** jpena is now known as jpena|off16:52
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: DO NOT MERGE
*** lucasagomes is now known as lucas-dinner16:55
*** bana_k has quit IRC16:59
openstackgerritwes hayutin proposed openstack/tripleo-quickstart: DO NOT MERGE
*** bana_k has joined #tripleo17:05
*** bana_k has quit IRC17:09
*** ayoung has joined #tripleo17:13
*** bana_k has joined #tripleo17:21
*** coolsvap has quit IRC17:21
slagleEmilienM: on to the ping test in the ha periodic job17:27
EmilienMslagle: I see it17:27
slagleEmilienM: the nonha however will fail17:27
slagleno valid host17:27
EmilienMMessage: No valid host was found17:27
*** amoralej is now known as amoralej|off17:27
slaglewe can recheck that if ha passes17:28
slaglebut let's wait17:28
EmilienMit smells good17:29
EmilienMslagle: I propose we retrigger the patch now to save time17:29
slagleEmilienM: wait17:29
slagleEmilienM: we need ha to finish17:30
EmilienMit doesn't matter i think17:30
EmilienMsince one of both will fail17:30
EmilienMpromotion won't happen, right?17:30
slagleit does matter17:30
EmilienMwe need both jobs green, no?17:30
slaglethe ha pass will still get registered17:30
EmilienMah I didn't know that17:30
EmilienMin the csv?17:30
EmilienMok now I understand how it works17:31
EmilienMok it runs the upload thing, so I guess it registers17:33
EmilienMslagle: isn't it a problem if changed between 2 jobs?17:34
EmilienMlike ok now we register ha job with 92/fc17:35
EmilienMbut what if the consistent URL changed at next run?17:35
slagleEmilienM: yea that would be a problem17:36
slagleEmilienM: so we should update the patch to pin to that repo17:37
slagleand in the job name we need to s/gate/periodic17:37
slaglejust noticed that17:37
slagleanyway, if nonha passes, we can fix that manually17:37
EmilienMgood idea17:38
EmilienMwith repohash=92/fc/92fc595cbc2186a8471987b93f6fb2d57912f565_e940f44317:38
EmilienMslagle: done, success17:38
EmilienMnot reported yet in Gerrit17:38
EmilienMbut it's in the csv, right?17:38
EmilienMI confirm consistent repo has changed17:41
EmilienMI compared themm17:41
EmilienMslagle: what is the url again where we can look csv?17:43
*** mgould has quit IRC17:50
*** iogg has quit IRC17:50
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
*** gfidente has quit IRC17:52
slagleEmilienM: on the rdo server? i'm not sure actually17:53
slaglei dont see it anymore17:53
*** pblaho has quit IRC17:53
EmilienMslagle: I don't undestand why you wanted to wait for the job to finish actually17:53
EmilienMbecause I don't see where it write the csv17:53
EmilienMwhere is the csv?17:53
slagleOk. let's reset17:53
slaglei dont know what csv you are talking about17:54
slaglethere is a file on the tripleo mirror server called metadata.txt where the successes are written17:54
EmilienMhow does it register?17:54
EmilienMwhere can we read it?17:54
slagleon the mirror server17:54
EmilienMis there a public URL?17:55
slaglelet me check17:55
EmilienMmy question was just if we had to wait for all jobs to finish or just ha job17:55
EmilienMbut it seems like registration happens within the the job so we don't wait for all jobs to finish17:55
*** jdob has quit IRC17:56
slagleEmilienM: they're under here
slagleit's organized by repo17:56
*** [1]cdearborn has joined #tripleo17:56
EmilienMok so we need gate-tripleo-ci-centos-7-ovb-nonha=SUCCESS17:57
EmilienMand promotion will happen I guess17:57
slagleso you can see
openstackgerritMerged openstack-infra/tripleo-ci: Reinstall python-requests in bootstrap script
slagleEmilienM: yes, except it needs to be s/gate/periodic17:57
slaglewhich i fixed in my patch17:57
slagleand if nonha passes, i will fix the metadata.txt manually so we get a promote17:57
EmilienMslagle: good17:58
*** jdob has joined #tripleo17:58
EmilienMslagle: sounds like good plan, thanks. And now I undertand how it works :-)17:58
*** fragatin_ has quit IRC18:02
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: Add parameter for enabling purging configuration files
beaglesEmilienM, when the smoke clears, what are your thoughts on proliferating purge_config throughout the service definitions that use relevant puppet modules with the purge_config parameter?18:08
openstackgerritMerged openstack-infra/tripleo-ci: Make $TRIPLEO_ROOT overrideable
openstackgerritMerged openstack-infra/tripleo-ci: Raise ping test count after multinode setup
beaglesthe mtu thing forced our hand for nova and neutron, but I bet it's bound to bite something else in the future18:09
beagles(if it isn't already and we just havent' figured it out yet)18:09
slagleis this why my nova.conf looks so sparse now? :)18:09
beaglesslagle, probably :)18:10
beaglesslagle, although tbh.. I noticed nova doing that before we merged the recent changes. I think it might be doing something else18:10
* beagles makes a note to look into that.. 18:11
dprincerbrady: +2 here, but I had a question for you inline
openstackgerritMerged openstack-infra/tripleo-ci: Fix resetting /etc/hosts
*** chem is now known as chem|off18:16
openstackgerritMerged openstack/tripleo-quickstart: use overcloud_templates_path for all template references
rbradydprince: the parameters are stored in a mistral environment in the variables attribute.  I have no control over the "variables" naming, this is a mistral thing.  Anything in variables is json encoded and stored in db18:21
dprincerbrady: oh right, thanks for jogging my memory18:21
rbradydprince: so the parameters are stored in variables['parameter_defaults']18:22
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
dprincerbrady: probably fine18:22
rbradydprince: I have no strong opinion on the naming of any of these things and if you feel there is something more descriptive, I have no issue changing anything18:22
rbradydprince: I'm still working on that patch at the moment to find the source of an odd behavior.18:23
dprincerbrady: okay, mark it as a WIP?18:23
rbradydprince: will do.18:23
dprincerbrady: I really wanted to try the CLI patch18:23
dprincerbrady: but it has a ways to go still18:23
rbradydprince: thanks for taking a look at the patches.  I think the Plan List and Plan Deletion actions are ready18:25
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Update for new HA arch
*** jdob has quit IRC18:27
*** jdob has joined #tripleo18:27
*** sarath has joined #tripleo18:27
dprinceshardy: I'd be happy to see this land as is I think
*** weshay is now known as weshay_mtg18:33
*** sarath has quit IRC18:34
shardydprince: sure, I'm fine with landing it, but when we wire in custom roles it'll need to be reworked18:43
dprinceshardy: yep, roger that18:45
shardyObviously it'll also break if anyone ever uses ResourceGroup for any other reason in overcloud.yaml, but we can figure that out if/when it happens18:45
beaglesshardy, re: - this is cherry-pick'ed backport in name only, really none of the changes translated, but I figured it could be okay if the commit message was annotated a bit18:46
*** jdob has quit IRC18:46
*** jdob has joined #tripleo18:47
shardybeagles: ack, sounds fine will review shortly18:48
beaglesshardy, yup.. just wanted to run it by you before you departed for the weekend18:48
beaglesthe commit/backport thing, not the review ... other's can do that if need be ;)18:49
EmilienMbeagles: sure thing18:49
EmilienMpabelanger: can you review pplease?18:49
beaglesshardy, besides it's probably fair for it pass CI before anybody weighs in there18:49
beaglesEmilienM, you are referring the purge_config thing?18:50
EmilienMbeagles: yes18:50
beaglesEmilienM, if so cool.. :)18:50
beagles(too much happening at once ;))18:50
*** TicToc has quit IRC18:54
*** tosky has quit IRC18:58
*** fragatina has joined #tripleo19:01
*** jprovazn has quit IRC19:03
*** fultonj has quit IRC19:04
*** fultonj has joined #tripleo19:04
openstackgerritAndreas Jaeger proposed openstack/puppet-pacemaker: Move other-requirements.txt to bindep.txt
trozetdsneddon: hi19:11
dsneddontrozet, Yo19:12
trozetdsneddon: an OPNFV user is hitting this: bug 1297535 in openstack-puppet-modules "Undercloud installation fails ::aodh::keystone::auth not found for instack" [Unspecified,Assigned] - Assigned to emacchi19:12
trozetdsneddon: did you ever figure out the exact cause?  He also has to use a proxy in his lab to allow undercloud to install correctly.19:12
trozetdsneddon: seems to do some kind of yum update/install19:13
*** jdob has quit IRC19:13
dsneddontrozet, I think this was a problem in upstream TripleO that was fixed some time ago. Is someone hitting it recently?19:14
trozetdsneddon: yeah I need to check which version of our RPMs he is using, but I thought he was on latest, which would mean latest stable/mitaka images produced by trown|outtypewww19:15
*** TicToc has joined #tripleo19:15
*** jdob has joined #tripleo19:15
trozetdsneddon: we don't see it at all in our CI, so I'm thinking it has something to do with the required proxy/yum failing19:15
dsneddontrozet, Your user is getting this error with downstream? I don't think the problem that caused me to open that bug was ever present in the downstream packages.19:16
trozetdsneddon: no it is RDO19:16
dsneddontrozet, Ah, OK, yeah, so latest stable delorean packages.19:16
trozetdsneddon: yah, what is the undercloud installation yum updating/installing?19:17
dsneddontrozet, I haven't attempted a stable/mitaka tripleo install in a long time, I just know that trunk doesn't have that problem anymore.19:17
trozetdsneddon: we don't want it to go to the internet at all19:17
trozetdsneddon: ah19:17
*** weshay_mtg is now known as weshay19:17
dsneddontrozet, I don't know if that is related to package installation on the overcloud image, or what.19:17
dsneddontrozet, But I think that the best thing might be a yum proxy, so it could go out and get any requested packages without a direct connection. That way the operator wouldn't have to ensure they had every relevant package mirrored.19:18
trozetdsneddon: in opnfv we advertise that the installation does not require internet, so we don't want the undercloud install going to internet at all19:19
trozetdsneddon: so i guess we need to figure out what is causing that19:19
dsneddontrozet, Yeah, well, I'm probably not the right person to ask, because I don't know what the undercloud is actually doing with the packages.19:19
*** milan has quit IRC19:20
trozetdsneddon: np just saw that bugzilla so thought I would ask.  Thanks anyway19:20
dsneddontrozet, Also, using "DIB_INSTALLTYPE_puppet_modules=source" will definitely require internet access.19:20
dsneddontrozet, Because it will need to use git19:20
trozetdsneddon: right it looks like all of those DIB_ISNTALLTYPE values are set to package19:22
slaglebeekneemech: are you here or just casual? :)19:22
beekneemechslagle: casual :-)19:22
dsneddontrozet, Although, looking closer at this ticket, this error didn't happen when the undercloud was downloading packages. This was actually a missing package during the undercloud installation steps,19:22
slaglebeekneemech: alright, well, i was thinking of stopping sahara services on rh1 oc19:23
slaglei mean really, what's the point. might help with cpu. they are occassionaly consuming 10%19:23
trozetdsneddon: my theory is, something is missing on undercloud, causing it to trigger install from internet19:23
beekneemechslagle: +1.  I already killed gnocchi there.19:23
dsneddontrozet, Yeah, I guess I could see that. Do you have log messages showing that a remote RPM install failed?19:24
slaglehold onto your butts19:24
trozetdsneddon: we have a meeting with the intel guys in a few minutes who are hitting it19:24
trozetdsneddon: we log the entire undercloud installation, so we should be able to find it19:24
*** rajinir has quit IRC19:25
dsneddontrozet, Cool, good luck with troubleshooting. In my case, the openstack-puppet-modules RPM didn't get installed (for whatever reason). So that might be something to look for.19:25
trozetdsneddon: thanks.  will let you know if it pertains to the same bugzilla19:27
*** fragatina has quit IRC19:28
*** fragatina has joined #tripleo19:29
*** fragatina has quit IRC19:29
openstackgerritJames Slagle proposed openstack/instack-undercloud: Enable sysctl nonlocal_bind
pandapabelanger: how do I re run doc jobs ?19:56
slagleEmilienM: i wrote yaml and ruby in the same patch ^19:58
beekneemechslagle: My deepest sympathies ;-)20:00
EmilienMslagle: I'm on something now but I suggest to use tripleo::profile::base::kernel20:01
*** radeks has joined #tripleo20:01
pabelangerpanda: recheck20:04
*** jdob has quit IRC20:11
*** radeks has quit IRC20:21
openstackgerritJames Slagle proposed openstack/instack-undercloud: Enable sysctl nonlocal_bind
*** iogg has joined #tripleo20:24
*** yamahata has quit IRC20:25
*** dprince has quit IRC20:25
*** rajinir has joined #tripleo20:27
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Create composable mapping between enabled services and role ips
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert ServiceNetMap to a nested template
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove deprecated node_ips hiera keys
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Rename NeutronServer to NeutronApi to align with ServiceNetMap
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add service_ips in csv format to allNodes hieradata
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move ceph cluster config out of allNodesConfig
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Generate composable service node_names lists
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo: Move ceilometer api to run under apache wsgi
beekneemechrh1 is about to get a workout... :-)20:29
slagleoh you're still here :)20:30
slaglebeekneemech: is there anyway to logon to the uc during a running job?20:30
slaglei guess you might have to be infra root for ssh access20:30
beekneemechslagle: Yeah, it doesn't look like we boot them with a key.20:31
beekneemechkey_name                             | -20:31
trozetdsneddon: ping?20:32
*** shardy has quit IRC20:32
dsneddontrozet, pong20:32
trozetdsneddon: it looks like there are some packages missing in undercloud image which is why it is going to the internet20:33
trozetdsneddon: but they got around it by configuring proxy to get hte packages20:33
openstackgerritMerged openstack/tripleo-common: Adds Plan List Action
dsneddontrozet, OK, that sounds like you can maybe work around it by downloading and installing the packages. That is worthy of an upstream bug on bugs.launchpad.net20:33
trozetdsneddon: now they are hitting Error: Could not prefetch keystone_service provider 'openstack': Command: 'openstack ["service", "list", "--quiet", "--format", "csv", "--lo20:34
trozetng"]' has been running for more than 20 seconds (tried 8, for a total of 170 seconds)20:34
*** jayg is now known as jayg|g0n320:34
dsneddontrozet, They should check to make sure that the keystone and other openstack services are running.20:34
trozetdsneddon: mysql looks ok20:34
dsneddontrozet, "openstack-service status"20:35
trozetdsneddon: yeah looking at htat20:36
trozetdsneddon: do you mind joining this call fora few minutes?20:38
vkmcpabelanger, hey there :) quick question about diskimage-builder... can you build debian images from a centos server?20:39
vkmcpabelanger, what dependency should I have to get debootstrap?20:39
dsneddonslagle, Can I get a review of  We need to include that downstream for an OSP 9 zero day fix.20:39
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: Add constraint to prohibit balance-tcp from BondInterfaceOvsOptions
slagledsneddon: it appears to be already merged20:40
pabelangervkmc: yes you can, last time I tested it20:41
slagleEmilienM: nonha job is not looking good20:41
EmilienMslagle: ovb?20:41
dsneddonslagle, Oh, I didn't realize bnemec had pushed it through, thanks.20:41
EmilienMthey re running20:41
pabelangervkmc: should be the only packages you need20:42
vkmcpabelanger, thanks20:42
pabelangerif you need more, I need to update my ansible playbook20:42
*** rhallisey has quit IRC20:43
EmilienMeven undercloud job is broken20:43
EmilienMand takes > 40 min?20:43
EmilienMundercloud sanity check passed20:44
slagleEmilienM: i think the uc job failed b/c of how i forced the periodic job20:44
EmilienMsomething fails *after*20:44
slagleit tried an overcloud-delete20:44
slaglewhich clearly aint going to pass20:44
EmilienMbut now why nonha doesn't pass20:45
slagledunno. it's stuck deploying the controller again20:45
slaglewill have to check logs when it fails20:45
EmilienMno logs
slagleEmilienM: the ovb job20:45
EmilienMslagle: ovb job hasn't failed20:46
EmilienMthey are both running atm afik20:46
slagleit's going to20:46
slagleit's been stuck on deploying controller for 30 minutes20:46
*** weshay is now known as weshay_afk20:47
EmilienMslagle: at this stage, we can retrigger them20:48
slagleyes i'd like to get logs from this and then we can recheck20:48
beaglesdid gerrit just go *plonk*20:50
bswartzbeagles: scheduled restart20:50
beaglesbswartz, ah20:51
-openstackstatus- NOTICE: The Gerrit service on is restarting for a scheduled upgrade, but should return to service momentarily:
bswartzbeagles: ^20:51
bswartzslow bot is slow20:51
* bswartz pokes openstackstatus with a sharp stick20:51
* beekneemech gives openstackstatus a carrot20:52
beekneemechDifference in philosophy :-D20:52
-openstackstatus- NOTICE: The Mediawiki service at will be offline from 21:00 UTC until approximately 23:00 UTC for a planned upgrade
*** ChanServ changes topic to "The Mediawiki service at will be offline from 21:00 UTC until approximately 23:00 UTC for a planned upgrade"21:00
*** fultonj has quit IRC21:01
*** shivrao has joined #tripleo21:04
*** lblanchard has quit IRC21:08
openstackgerritJames Slagle proposed openstack/instack-undercloud: Enable sysctl nonlocal_bind
*** fragatina has joined #tripleo21:11
*** iogg has quit IRC21:11
*** yamahata has joined #tripleo21:16
*** fragatina has quit IRC21:19
TicTocCan anyone explain "tripleo quick start ~/.quickstart/ssh.config.ansible will try to proxy through the localhost to ssh to the localhost and will cause an error if ssh is not setup to support it" how do you proxy through localhost to ssh?  keep getting the ssh error connecting to undercloud though I can manually connect to it21:24
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart: Expand templated bash to allow for doc generation
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
*** myoung is now known as myoung|biab21:38
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Deploy minimal services in multinode job
*** toymachine has quit IRC21:46
*** jistr has quit IRC21:47
*** fragatina has joined #tripleo21:48
*** fragatina has quit IRC21:50
*** fragatina has joined #tripleo21:50
*** jistr has joined #tripleo21:51
*** pradk has quit IRC22:00
*** jistr has quit IRC22:00
*** shivrao has quit IRC22:04
*** lblanchard has joined #tripleo22:08
*** jistr has joined #tripleo22:09
*** shivrao has joined #tripleo22:10
*** beekneemech is now known as bnemec-pto22:12
*** bfournie has quit IRC22:14
*** [1]cdearborn has quit IRC22:16
*** lynxman has quit IRC22:18
*** jistr has quit IRC22:19
*** jistr has joined #tripleo22:25
*** TicToc has quit IRC22:27
*** panda has quit IRC22:31
*** TicToc has joined #tripleo22:34
*** lblanchard has quit IRC22:38
*** maeca1 has quit IRC22:39
*** lynxman has joined #tripleo22:43
openstackgerritMerged openstack-infra/tripleo-ci: Don't try to get stack details if the stack doesn't exist
*** apetrich has quit IRC22:54
-openstackstatus- NOTICE: ok is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong23:03
*** ChanServ changes topic to "ok is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong"23:03
*** ChanServ changes topic to "TripleO : | | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt"23:09
-openstackstatus- NOTICE: is now running Mediawiki 1.27.0; please let us know in #openstack-infra if anything seems wrong23:09
*** bfournie has joined #tripleo23:13
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates: Remove NeutronEnableTunnelling from templates
ayoungtrown|outtypewww, larsks if either of you are here: what do I need to do to deploy OSP8 (or liberty) with TripleO Quickstart?23:21
openstackgerritMerged openstack/tripleo-quickstart: Only set $LOCATION if it is not already defined
*** chem|off has quit IRC23:22
*** Ryjedo_ has joined #tripleo23:25
*** Goneri has quit IRC23:26
*** saneax_AFK is now known as saneax23:26
*** saneax is now known as Guest4826923:27
*** chem|off has joined #tripleo23:27
*** Ryjedo has quit IRC23:29
*** Ryjedo_ is now known as Ryjedo23:29
*** rlandy has quit IRC23:29
*** trozet has quit IRC23:31
*** trozet has joined #tripleo23:32
*** shivrao has quit IRC23:32
*** TicToc has quit IRC23:34
*** TicToc has joined #tripleo23:36
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: DO NOT MERGE - Periodic test
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Deploy minimal services in multinode job
*** hparekh has quit IRC23:58
*** yamahata has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!